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[3KB] aib*-** ^ r*tt»ic!ti;t:iww:«»^ 
ram? & 0 

A /j U a 5 x. 6 tote ht> t - * 4 • 6 ES!f58? r*^f£ 
SfciitriEAAf^fc J: OTttft^RtC £HriaWfj§ <h <7> 



£ a s « <t- e 
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1 

xtm&&fi?zftzB£m&*si.>x. 

At>mT<>u x^&oA:bflM*fHRU zowmcg 
coD^££ft*mfgf^ft«€-ntc. ztmoyamcfcL', 

mz ht> h man o ^ k h n £ matt 

♦^^SBiRf HSSAJ7^ r < 7«*c#fe£Sft* 

6<*>tiMR* iS«r J: ft 6 £*>1* 

tc. kHi-^^cwaafwiaA«>s*Btt»tcai». aa 

m^xtt\m%i%&£.2 ****** ± > 

ttmr* * ^ ^ 6 a * tittle* . 

me ;^*r* 6»ra»f&**^ a ^ft-smiiaa 

fulfil ffi$@rsa a r ^ 7^tc^T-5|ijia^if is<?> 

rccfij^^^T^ > 7 * rtm^&t . 



^¥9-9 1112 

2 

c^^$ftyc*tJc5t#fR-eft-t-*ncc. gfs^ewecj&r/ 

Km < << &t)^&<L . 

nm%ft>h$>%. < <t <>— oodaa^w ^tc#A6ftfc 
A/5 ^ fiinaBajssft^ie^® aa 

^^ftij^%-ri«iaAAJi^^A<7>^*s, fij 

J: ^A^^If^^gfgi , 
liisap-5Jg«g^Scc#^6ft*AA««*, A/J>r ^ 
r^^^T^Pia^m$R<7>^^6ig^^>^, fe 

<7>xj( xm&& £*>i*X'*>2> m la a s A > r> r * fl^* >m 
[ l£*Jf 5 ] ^la^^e^^tc^ . AAKEte J: o' 



http://www4.ipdl.ncipi.go jp/tjcontenttrns.ipdl?N0000=2l&N0400=image/gif&N040l=/N... 2/23/2007 



r ^ rm^mcicmm^mzz&vzt . as 

m%&c . m&mzwm^mc uAftmM^ 
&^±u, mid\^smm^nm^/^ l<x ^ ^ 

£ n x i > 7 v 7 $m ^u^> xnm^c **r z> 

i>tmz&t>?zm&mikm&icmm^m*&bxAft 
Kmc* o &jj?&mmw<>citt>x . 

w>*>, aw ft* r < 7 tm CCSfc o X$>t* < 1 4>—^> 

nmcu-? reform t> < A KiP 6^/jT6^^^ t s 

fiLtcm&mk'4mv>$:ii £ $mm . 
hft^m-> h mmmm&^z £ n * m 

<D®$mtti*x$>z>A&tis r < rtmzmmmm 



(3) ^¥9-9 1112 

4 

t. 

*&znmuxh/jitif£mzz&±$i*. co^^nfe 
10 \*>x. 

^^6a>lilR^?#^/c^pnf^S^A^ffi^W 
a^H, Aih/j^r^7$m^U^X^>U< £h--? 

mw^^^xwi^mmm^M^hm^ntcim 

m&m&mfrgw&h<D&m&mcfcox Aft* ?4 r 
m^. tbftsr <7mm<c>£m*mz^u, «fc t m 
20 m\Tj^mtmTj^mcj:zAftfizw&?zwi%wm 

f<J^^ < t t>-~o<DA?> r; <A^c^\ hfttc 
A/j $ i»h ffiimmSik&SsitcZ k Z> AA^fttttR 

[£g*Jf 9 ] mi* r < 7i!^^i<riiAS/5>r> 
40 r$y^*ISai'r^> > y 1 >7fiJ^I2tft^n^ h^^*>, fj 

7ii^i2teAa y hccHi^is^^nrc^AS 
v 7 pmft&zmm o x&tctt * r < 7mm *tf <t ^ m 



http://www4Jpdl.ncipi.gojp/tjcontenttrns.ipdl?N0000=21&N0400=image/gif&^ 



2/23/2007 



(4) 



1 1 ! 2 



iBt^hJistt^o? w\u*t <<7p\St 

a h 'jie^ad ? v icztixxicmmztix &tc\&fi 
m ? < Tttmv&n&msc, Affi*?> r * riii^g 

rprnzmz* f << r^m t a mj taisA d ? h tc&m 

fci* 3 3 fc it 4 * & li 7 * fcii 8 c i *iatt<o^ 
[is*f ! l J Ma#B«a^RKi*. WSWWHft 

jwse i/ , *tts«a*a«: *f u tra«*a*ai;-c»(* 

cc A /j 1/ r 6 5 J: ^> Pie A^^RK A AS** 

3i» * £^ a*i*«*#fe* u trnsam #s«ra c -c & 
ttu. raffi^!i^tt4R«NraeftitBA s ait¥S«a 

«.VCJ3f*OT< 

a«>IBm*«tift«Ji, msAA«#1Mft. GU<ii|tj 

^^tfTMa^fTnr^mtt^a^&c^^wft^T&tt 30 
^1 a eato-?* * - 

fE>oi5$E£f#& tcfromu ssa Affi < * 

A CO 5 , Atti/W r * 7 tm left-? X&t* < £ fc— 

muifffl&K&x ^ ? T^cpfeafiteKft o x 

t '< 7S!ii0>£SS«4a&»^u 5^c. MBAA^T-7 



t z < ^ x £ aw ^- ^ (Q&t>75&<om&i}i*<C'3>u 
cn^AttiA^^* TSis^ssaAaojg^s 

[ ■:) •:) 0 ! ] 

c^c . ffl**>*-cfw^ittJaa*©AH«j*R«:iir ^ 

[ 0 0 0 2 ] 

A. -7^i?p^*>, *??'(4jU. ^'-i^^t 

V i'^a^^^fe^ A* Tttt> A/j~7s < 
tevttmZZAttZCt&X'ZZJ:'} 

*o fo« * a«s»-ca*T * ««ia u x * r 

feO. cn^SfSAffl^^WA^^tcfiJ^Ufci: 

♦J. |fil^C*t **-^> • -7-»^> £7 ^ ^ X<Q&& 

A ^ < A *fB*d*>1i*:ME v7^/5^ pTIB A/3* 
S. t > «t>feA^f cr 'f 7«tt*U r'iAvu 
^^b-^ £'<*>fe»cr>Uy? r < A ^ffj^ 0 C , M 

v A ^ u ^ 1/ tc AMOflOXMf i , 

co^m^r^^rvA^e^^i^^^c^cc:. sia^ 

S^^WA^^^ai^R^^cta^^^^^A: 

x a Mm<omm^^^ut>txx^^ 

[ 0 0 0 3 ] iC>St'St*. AA-A't'lgV. 

fc*fis*i&jtc:g^t >xttm*y >&mmo. mis^bucc 
tei»^R^&nfcAaj/7#tt^ffl^**c^.'i?-*-yju 

[0004] 
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7 

StCtf, Ems 4 X&Bfr** 

I o o o 5 ] «si*#*^c aat*c io 

n&<otffi v&%jsmk<Dffi%&+&ti& u x ir *j& c * fc 
«>. *j«#u:j:->-ct*ffl«tiajo«a*»*i/t:4>asafc 

[ o o o e ] & 4>n or. »©7* ? * - v>vmm 

Si* CO J: ^Jawa#Kl*tOTfe, ft*fCJ:&A2M>l/ 
0. M'XfcAOattfcO^il'SWEfl* 20 

[ 0 0 0 7 ] 2*, fltofctt^K J: ftttSAAKfttl/t; 

AtiD-r * * - 50t#ttmcc ^ C i*Affl^> ^> T 30 

» i {r- 3 > £l»X?*0ftWtt'Cft -7 fee 
[ 0 0 0 8 ] #3 , fiJfflS^ f »J jl 5£A,C,J tio 

C ft *>0lx if. R*r AAMR? 0 fc *J 

Tft^, a * , . x >- - >> 5 >Sf£AS|U*« pftttt^SA 
rrt-CA^KiisMM^&'r. c«>J:$JttASW?-**w r 

[ o o o 9 ] «^t*faE^6**«ss4a«:fei- 
vfcBS, UAJBSKTO^n $ > ^SftfcKftcB 

Wf*©ft*tt*#ab*fl>«:. *-ft£ — :>— ^fctr-ctt* 

fcOii^fc. *y^*tc<Lort>^D^< <<£w<:< 
AfcB/jt^ <t a 5«wv»&r* -> fco so 



-9 1112 

tc*> . t's\ t u<onm%& ^xr u&ikm L,&He I, fc s 
¥ : < rif^^oXAffi^^i^cift^fjr^^^^^ 

i r ffic >AJ7#ftt*& Aftii u r c - & fo« via 
mil *4f«J^**^cc^^AA«t^, 

ft? t? ^ i,* -7 A^tc , mm^tc i: o igc »a c » aw**» 

[0 0 ! 2 ] St^O^iS^StC^C^Tii 
Cft60igiA©sP^#iS$ftt:fe*>'r i B*W<fA 

ffl**S^>jfi&^*i«5ftrc*t:. ««*€[>»*t?A 

[ 0 0 13] ^ "c, A^^Rfrffibtt^tBfEO-?^^^ 
a^feCi^. i'OA^^R^^^tt^rflWBlte^ 

[0 0 1 4 ] Rte, Mx.m£&<^&lZ£Z>Aij ; bl<< 

&hijfo&i>&&t)<>c.m^z>t*t\ Ernie*? < rro 
[ o o is] ca£*z>^^- ^ji^hs^s 

"C'i* . mm%& >^ r ^> ft 

fc A * - K CC^fi^i* ^ ffi ^ r V ^ ^ A £ ^ ?t 5 

> $ ? * 4 s\xW*is>-7icttih, nm%&tis%L<D\ftic 

X 4<5 0 , i * (C i^ta A©*tt^JS@ i *> U *, ^fco 
[ 0 0 16] ^^lilSS^:- FTUSWtftettiC 

jw&momsx-bs mm£uxttzz<Dm&&*&f&t 

Ur>X U£-?X<<>tCo 

[0017] -ecr, ccc^^g^cL-r^ic^ii, 
f*j^^ft c * v^ai * > * ? * a x t u o f# £ .'U 

rv*. *£*ja-cattfrAHy7«>y ^ < rii^^^tc 

I 0 0 18] 
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**I*S«ltf AA:«sfelB*^- *&ig«"T**S: 

tc > X mm?* ¥mo>& =t?£ WX' r&M 1/ 5^1£ £ at* 

[0 0 19] *SMicj:*w*. f«*tfSt#»i>*7A 
t **!S£?f 5 IStfC , #ts<55 «#£l?«fli»Jttft A 

Site 1/ & Atttfrfra££tt o , * icvm%wzv>n 
rnzic t r> x%m i< w<< > a as a t * fc a^<?> 

I 0 0 2 0 ] £ <*:ni* . >f^rrD ir- u 

w<: (to r Aitt^RGXa ^1* «:3«R o ^ss^ 

icim^»a^ t« ? * - $oi>**is^g 

O^IS^r^r -5 C i ^5]g| i ft 

[002 i ] mic^mniz Jiftw, t^o^ti?*-? 
} i> ** ig-gis ft tfft ^<e>*usi* tcfcf ucw^k: j: £ a ^ 

o> i/ a i/ ^isis ^>3?* i/c u * ^ <f 6- . mmo>m#&£. 

U toff* ^fRfrJMR? * C <t CC<*; 0 C <^>r»0 fcjgft 
1/ . WJ!*a*B W<&JM**> * M^A & C <L ft < St 30 

[ 0 0 2 2] »«C*»W«: Atitf, A/5<^>«I«R&S^* 

[ 0 023] m^icm&&hhzfts*mmiz^> 
b*<zfamim<D&fi&%*x&*±if*tj:x. tb^^ 

[ 0 0 2 4] ^^tT'li»W^A8l^^S*jg 

r 0 < < ec < c » Affi^^i * ^> ct 5 



^¥9-9 1112 

10 

VZCtlCJ:*}. *'J^#«lt>^0^> 

[ 0 0 2 5 ] M*c j:n«. f^O^ljS^gtC 

fee li<St ^Affl^^tcBB-r 18 ^ AO^RA^ 

[0 02 6] KK. -£m*l££tilZ. AA^ffi€:S^I# 
R*«C* *^ * £0>A/j^&l&*> %V*f=kX*m pJfe 

a«:*it/'c, *uffl*^A/5^«fflT^c i^pjftja#s 
^> o < ^^m^^#-r ^> c i tc ^: 0 c c> 

A*h]& i ft »J . *JJ^*Kfo^ 0 W < fit - ^>TC W > ^? 
[ 0 0 2 7] 3* <t/t, *U8i#t*f<Wf« 4T7" «J dr - U 

> ^ > ^ -7 * 4 * & ^ C £ A*bJ&K ft 
[ 0 0 2 8] 

^ciocj:^. KK(ci&t;/cAffiA^«f<fJi4«:«« 
mfl5lteftA**R«r«fl3acc*D^t/^^R«:^«r 
$6<CAtBA^«(Offl*&fc1i*5SaibT««Sia>^ 

[ 0 0 2 9 ] m \ it&m*<og,ftMiz%z>^&?*~ v 
>imm&oMfmx'*>z. m 1 ul^^^k.^^o> 

^^^-^iaigl Oii^JjtH I, St- < Til 
^&aEI$ 1 2 . WasWlS 1 3 t S^gP 1 4 coif 4-?<?> 

[0030] A/3^1 H*7-?ArV<^A > ^OD7 
^^^7", ^^wi^AAf'^A f DVa - 

^ =£^fe^>ai7^r ^> WAODVa-ODVn ^©Ktt 
[ 0 03 !]/f> r Sl^rx*^ ! 2 %*AAfflttl/J^co 
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xi 

■it. Trttt**. *7<*T$m < 7?atr-fs 

[ 0 0 3 2] S Mfg^lgas 1 3 ttlWBfe^lg 1 5 

4 ^^«!>«aMff*wtc«^i»r r < 7ii^<Dp*<Hi 

I 0 0 3 3] F-ra*§ft~« 1 5 l*> AA?CC^l,'C-e^ 10 

$ 7 w f » vtras, cn«*«scc 
r ^ i» * - v ? > r*> o c laj* i*, ig^rt ^ a t> 

[0 03 4] ( A^l 1 fcJrO'ffi/jSIU 4<Z>If$ffl> ID 

KABStf 1 ! 2. ' < ?®M& 1 ! 3<03^^>^e 20 

r^IS^P 14 1. tfc^*:- K4(Liai^ 14 2, ft/J^X-V 
[0035] KAtt— 0©^<^^3^fe«E> 

[ 0 0 3 6] latflK, Xf<7i tiffk {-ofi J: 30 

[0037] PUtt, - *Wx*7tm*-^Pz 

a -7 > K >' r < 7 \t , .» < V ^ >^<7># ^rv> ^v 1 -' < 
x T' £ & v * r < <Y * £ OA/j £ , $ «c £ & =2 

♦j . 77*) ^--^3 >^^-7> r^fg^r 

[ 0 0 3 8] < \t>2& 1 1 > A^& 1 1 G>«.#BSr*> 
£A^f^W*!&5l^! 1 Hi, £43tfgAA? r ' > <'f 
£f£ig £ nfc £ & A /j <L O , r < V'sfr-r- 5^<D 

A*^>WX»Sfc D B«:mi**^CC^o 0 ^.»W^«43a 50 



^¥9-9 1112 

Z>. S^ti 4 ttfc? Z X^<0 Aft 

A*-K«8W1 l2tCisf1^^o Wit«^^^^<^ 

U\ 1 2tcffi^T^o 

[ 0 0 3 9] A/5^e- K»JS«5 1 1 2**, A** 
- K^A^^^lSlt'^SPS*1#.^PStcf#"?. — ^<7> 

[ 0 0 4 0 ] SAA-t- 1 1 2 T'jgttSftA:** 

m^. AA^^€T4&atti i3tc^(i$n^>. phx. 

I 0 0 4 1 ] KK* 9 '<< 79HWU 1 1 Zit, A/5^&- K 
«LfiS51 1 2&'6®&ftfc*JJ^*<DA£<&*^ K^<Di? 

ki&RUjtr <7ti>t>V>\i3&**r : 2 

[ 0 0 4 2] A/5-** r ^ 7»SISB 1 1 30<ft.«»ai*, 
^1*^^' r * 7mcj*1^2 tltcAt)* ^4 7**mb& 1 

[ 0 043] > rigj^^ ^ * * s wti, ^is^isi^ 

1 3 <D\%ihiC\t£K ^ < - i?y?7> Y7y ij *r - S,/ 5 > CC 

Si^v*, ^ h^i^- KAAABE*&& 

^Aj^^r/^v K-e- KA^^SgfIq D p0>S/3^A^ 
ib. <-n^^^^t)i*^> , 'r^7i(.,'CCDfi?^^ 
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[ o 044] 1 4 1 4o>m®,mmv$> 

3 a . ftTVx < r^esfi 14 3b. >r w r * K 

-{>^7 i ^^l43 c(£'3^tf>^> J .-jU^6s£&o 

$ n * r - * £ am o , -e mm > ?w t 5>yssp« D B tc 

[ 0 0 4 5 ] tttfM r v 7«MA 14 31*, 
it/TJ#fe> > f ^ rlf) ,W a ^ v» ? 1 4 3 a ^ J: 

mm > r * r<£K o n fc ffl a ^ - * * , *<o> -r 

- hV>^7x^ i 4 3 cii, «*MMWj*-K1» 

[ 0 0 4 6] KfeiE* 1 4 ^fifctfiA* 

KSfl>i« * 7^ £ fctfi *r v -I * SKBKrT <£ * - 
K • *1 4 2 b£ttJft«RKft 

X 1 4 2 I) tCf.(^t 0 

[ ■:> 0 4 7 ] *- K ■ ~T''< 4 * A > £ ? * X l 4 2 b 
CD — ?4>flUM* — ?£>ffi a? ^ < 4 xgfJ^tcMJtD U . <~<D 

~?Z>y -r a T • f ^ a ^ > ^ 7 ^ ^ ^o>SA^- ^ ^ 
[ 0 0 4 8] ( > r rSU^ft^^5 1 2<DS£SB> @ 1 5 



(8) <%m¥9 - 9 1 1 1 2 

14 

O^^-rri^Mttl 2v^, Ir^l 2 ifcttUC^l/ 

i:C>lf^ 1 2 3 £m,T^'f5&> , r : < 7il^iaiS^a ■? 
K 2 4€:WUt:^T*- K - > ^ ^ 7»^»PS 

r <7^^SUI 2 It*. ^tC7^y^-^?>tcl 

10 , ft *C*fc§ V£» 1 3 OF*-^ uc^tr^J^ iv 

[ o 049] <wsafsigp i sotmy m \ 5^^$n 

1 3 ti. A77>* r y 73/Lil^ 1 1 3 toiVUfiJ 
r < 7*m® 1 4 3^^f#6n^Adi/j^- KCOim 

>> i 5 t<or-5$Mv:'ttm$:, mwsw^at 1 5^ 

[ 0 0 5 0 1 * tr. ><^tTj&«KiM»r A 

73>^>r^is|ii 1 3<httlA-rfri 7^si^i 43, 

[0 05 I ] C«>J:^ft«ffi*fT^ttiail/rfi*. A77 
30 ^^ ^r^gpi 1 3^tii73^r V7^^1 4 3, 
j^fi??^^ (^^y7^>KT7'J^i/3>> 1 
5. tecfcO'^ < T^S&XE^i 2<Lc^fcfeCC.^^^, 

W5-2 166 1 8^|g?ni/feW*Ml5^^^^<0 
J: WKlV«9 (r ^^fflt^c'Att^> 1 Sacerd 
ot i, E. D. # h AS t rue tur e for 
Piano and Behavior' , Am en 
can E! servierfl, 1 9 7 7'^cc&<£ J: ^ 

40 

[ 0 0 5 2 ] tts£&il£fS 1 3^^ 1 3 1 ^rrttT^O 

^1S^5>^ 1 3 ! 132, 
cfc^^ag^ ^ 5 7^ ^ 1 3 3 <7> 2 ^CDsEf*. 

[ 0 0 5 3 ] U±&'*BM<Dfe&t*<0meV&Z o ^ 
•£Eg<D*1£ * ^ , - x-*> Zmggm^ ! 3 ^(^7 
50 P-? + - 9CC^T 0 C0H«:ll^'t**fi 
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[0 05 4] ST, **»M»*Sft*i, Mismis^ 
[ 0 0 5 5 ] >fittMB©— -WWBt*. * ^ - K i 

McftASft. k i d*p ? Kct*»-^^>y- K 
I o o 5 6 j tftss* a y v \c\t, > 

[ 0 0 5 7 ] P ? h fCti % P ? V 

V-F i DAP v VO&ifip 0 0 0 tT^^^ "7 
>/-K' ^Ci^:, ■*77>p00 0V 

[ 0 0 5 s ) &zmm&z*<DV7'mmcm.mT&tc 
&>ic m^mm 3it^imnm\s.^^n \ 32* 

MS§»»JS2tfe^-^ 1 3 2 K tell 2 CC^Ti: 5 

1 £ mm s - k i — ismbu ✓ - k t* 
id-. ~s«r . ~^&fr , -vzrmte 

y*h" O4O0C'AP ? h ""Jl-JH D" ti 30 

■y'^^y- K©->- k 1 d ilsltiK . s»i§»riJK ^ ~ 

[ 0 0 5 9 ] g*ft?MB>J y - Ktcii. ^MB"]***^ 

[ 0 0 6 0 ] "7 >^ 1 3 I ttt*EMMS2tt^-:7 
JH 3 2^^6nA:MiS^&J^±>0'6)^c^b, %l 

[006 1 ] -eor, s^wtc»asgts!^sis3nfc*i 

I*. rt^'U j D*P * hflDtf* fc r 0 0' i:'*>*4*s§M 

It r 0 0" tsZ&TZo 

I 0 0 6 2 ] m^'v >tl3! i*&£H#s?X'<D^St£> 



16 

<Dfc'iviii<ar<&*as§:/'j 5 -7 ^ ^tafti/fcWB^y 
[ 0 0 6 3] mz? »j ^ * < ^lats^-v-i 1 3 3<?>- 

t ? M D' . . *(±^^ <C>3KDXU ? 

7za\*>-r-7& 1 3 3 &<o. fo&mzzf »J ^ ~r < ?v - 

[ 0 064] m&nm& 1 3 Htm omr? <j> a - 

[ 0 0 6 5 ] VJMs f7 T niDAP^(^ 
ft^ ' f 0 0 " XihhMi^*) a r < 7V - K j cc-C i 

[ 0 0 6 6] > - ^tfB#cc^>^^r- 

ea^BfcofctB^, Mig^>7"i 3 nt2>fa- 

m>Ji&X\ v > hCDJSWij&SCEMjRO i«SS« 

[0067]r^t>^, ^-s^i§e^P, ^tfecift 

u»wuiTc\t*im7v i^Mup, tabu.? 
[0068] p, K&m^&K?^x<Dm*&mte£o' 

[0 06 9] -*?wT*£»*« 12a, ^>f^ 
TIU^ (MA) ©M^ti-C** fc MAH«' ic^^f 



http://www4.ipdl.ncipi.go jp/tjcontenttrns.ipdl?N0000=21&N0400=image/gif&N0401=/N... 2/23/2007 



17 

[ 0 0 7 0 ] * > * TttSffi**- - 1 2 2 CO— 
g ^ < r if^ss tR s-YL WO\ &X > r ■/ 

r«aa«-/ - n* v-KiD' . -Ji->f < 7ii 
S" . . "-x^ ^ < r sis' 4 a d - r 

[0 0 7 ! ] >^?rD^-S/ s >t«*l 2 lit, 

JI> 7w Til^iatfcA d ? h<7>rtsi > r a r%\Wbfc 10 
SflHRy-F*. -X^r < TWSMR^-^JH 2 2<?> 
[ 0 0 7 2 ] r v 7SflS*^SP 1 2 1 1*. ^11/ 

v>Y¥gft>t>, H7cc^f «fc^ft-t- K->*r '^rSU^M 

[ 0 073] jt ?£m&&& 1 2 1 r < rmm 

«*T r < 7&m&t F - ^ ^ < Til 

✓ - K<z>M&Aa - Kcia-^$nrc^^r >Tf(i^^ 

[ 0 074] zcqstj 30 

rnsMfr^ftamcc^y 7ii=^ origin, 
^i£^^>^ 1 3 1 cciEj^^n^o ccr, sr<rm 

^<Tlia3»R^-^«:J5MW*BS5<C % £>&>?w7 
il=*a FOig> ^ TSU^A D * h *> < te&f±A D 

[ 0 0 7 5 ] mi*, 3I> 7w Til^ A P * t (Qfc&ifi 

IB j.S A P ? ^ <D&&&lX& r> X i>ikfe A P h ODFTSi 40 
^ - F - -rf -r < 7 tt&Wk<D&&)A P •;> h <Ofyg-W&£ O 

F©*^ r ^ 7W"iS ad-; h *j J: t>'^:> r ^ Tf U^a p 
? h e<rt*, >* r < T&m&rmbWz Xfi* r < r t & 

fisr<T<D®<D J &X't>&&X'it$>. 
[0076]**:. C(OA/3>r^7i^^f < T(0 

Hj6^^-ri DAD ; h^'lt^ ~m0 6^ r-*^x>* 
•^i 7*a3BR-/- F«C*ji*r, JI^^'^Til^AD v 
HB*s - xxx) ' r-^D, ;x>^ <Til so 



^¥9-9 1112 

is 

K^fii^ fc {*<^jt?m, XXX> " i^^r 
t. -rvfc>fc. ^saft^sntr^iA*^^^ 
[007 7 ] c^-i*, v TSUiiAa ? hc-ffi^ 

[ o o 7 8 J « or, ;x> ^> Til^AP ? K^jv^ 
ffift^Mffii. ^--F-^^VTil^^cDAttl^^ 

[ 0 0 7 9 ] > T^$3&& 1 2 1 T < Til 

- tCD^K/- K i Dap 7 bo>fZ#> "mO ! " ^6 

[ o o 8 o ] we* t * 7%m**m \ 2 1 i±y - f i 

Dap ^ h <ofi^ ' m 0 6 - 0>Xst *r < Tilgis^ ^ - 
F^tt&gnfrC^^s^S^ttSO J: ^ i^^^. C^X 
^ X < T il^iMlR ^ - KCO^ x ^ 7 SO^ A P 9 h 0>*i 

1 0 081} -e <otcv>ic. > r ^ r«yaj£iaa5 1211* 

07 *BS*o. ^- F I DAP ? t-«:<fg^ fc mm0 1 " 
r^^-^e- F-^^^rnaaJifi^- Ko^f i 7iiS 

AO v h <?>tt*^^ r Y 7fU^5Et^A O y h (^fgill t> 
il^fc;^- FC-Affi^^- F AO 7 h^H^^^V 

[ 0 0 8 2] ^ ^> TSf^feWB ! 2 1 l*CO)*«*^ 
l*. ^- F ! DOffe^mO 6X'*>2>\$<D'ZS-T < 7il^ 

- i%^m. xxx) - im-ifc^so, &&xxx 

[ 0 0 8 3] COftMiR ^ < 7ilS^^- F 
^^^ YTSU^AP MCj^^^ti. C^aD-;^ 

tc. > r a r«a*.«« i 2 i tiiu ^ ^ 7f ij^ig 
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•1*11 u cwidittWrr*. s^k-**^ r«££j« 
ait^- ko^^se-t^o 

[ 0 0 8 4 ] KK*G*^^ft3ftfcI§d. r * 

r tiaflws««««i u , c <z>* - k - > ^ * r«s*h6 
h. xtcM^z riDxo ? t '-1111110 0" r«> 

K - ->f ^ < 7 ilStftfV - Km n* 0 0 d:S2£&r 20 
[ •:> 0 8 5 ] [*IW»ffW (»— ©«ff«> ) *^ 

T^Vir- Uv^ifi* T&frfc, ^SWl^'CfilO 
[ 0 0 8 6] 1 3«>*Hg«»Batft^- 1 

xo -? Met*. -JttHr^fs^ ic^aitseftw^ 

[ 0 0 8 7 ] *r*«S^.fift$H*fc^ Mf§^>^ 

uc. »a^>^ 1 3 1 kj: -?r*tES»fiij - r o o - 
[ 0 0 8 8 ] MB^ >-M 3 1 W^JHW^BIBT 1 *) 
n. s&tcftBMM ir «?n^ t^ig 40 

~MAtSli£1t i&m, MA l > OKffKfefcO. ^ 

Ms*— ^ i 3 3 ^l#^-r^>o 
[ o 0 8 9] *®ttt, wis:? ? >+ i 3 i \t. mz? 

[ 0 0 9 0 ] 7v T 1 2 tf\ -t-^j^O^ 50 



#fPHF9-9 1112 

20 

r < 7«s t mm*\w* > r <£>«k<Di*esz £ ^> ^ 

[ 0 0 9 1] #K>* r ^ 7$5^^SU 1 2 it, ^ 7 ^ 

[ 0 0 9 2] mZ'gW® 1 3 ti, ^ < Tiia?^^ 1 
2 (./ r * r < 7 fiJ^ ^ , AA > ^> T 
WStt! 1 3*Jj:CXffljft>^v T3AfB«pi 4 ^-^ 
•WiWfflT^*^f e: -/r*S*irr'6. cnccj: 0 . A# 

-rf^ < r^gjj 1 1 3 fe^^s^^^ < T*mm 1 4 3 
t« (ornate ^t^e-K-> i f>7 su^m.^ ^ i 

[ 0 0 9 3] C <omv>im > r ^ r fl£4>iS£. ^ 

^^2 o^:^^> ■^-fe-^^iiis^tg^'r^o 
W±c-4Aiaiu:^o, ^^ig? 'pi r* t^j^t-. Msg 
•y^^^ 1 3 l t^x<oi?-^'SS ~MtiS8r **tfcc^ 

[ o o 9 4 ] c (o t * , *f f£gmas> \ 3 rt<?>^ >e« 
[ 0 0 9 5 3 ^m<owi^nm^iL^L/^^^i>^ 

mz??>t 1 3 1 tt&zrmm fc MA^»r 
1 0 0 9 6] > ^> rif^?^^ 1 2 ^ 4 r 

[ 0 0 9 7] C C'.^, C CC-^iS^^Ji - b ^. - -7 

y l c >5 j ^ < rmm&miR $ n. *ri§«3aa 1 3 
isssn* 0 .*«s*a» 1 3i*ig^5^fc^^<Tiis 
tehus? 4 79*m$&\ 1 3±&fi*r4T9*mmi 4 
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[0098] C<D*r w7f@ "fa*— *<,^tf> 
^Ci T*S«i: £ 6 g **Ss:W* h > r ^ 7 log T 

Sir ^c?i r < ft* c t tc ti ~ x , <£ D AH**fc*> »J 1 

[ o 0 9 9] py^i*±<DJ: ^ v*I^i<:4<5^ 10 

r. A^£ftP**i*fc*?»K*^Afrfc~ Kt>u<i** 
i^di/}*- h$t^3^tt?ft&*A:tM'7 r <r rami® 

i i 3 6i/<k*HiA-rfr < r^iaai 4 3^>f^r 

[ 0 1 0 0 ] OX, *»W«*©vjU?-*-4r.ii 

0. CMC J: 9. ^KIfc(;TJ:4iet>4U'<f>*7x 
[0 10 1] W-fcti, ^^^^-^^ISMS^W^K- 

#fc itft?*4>4>'ca fl^accflW^* ©ABB 

iRf^ft^if^ C * fcT * &<> 

[o 102] ***M*«ctoi*T:. Wi* 

A WA-^r < fee-. <£Jiufc:<,>{>© 

f>fc*BrcAJj*tf^ C *3*C* 

£C *#ejfe£#£ 0 40 
[ 0 10 3] PUtt, ±|g<3C«ff««?>r TttS 
cc4<?<,>rt*, B 1 6<DJ:5*<:#;ii#-7H' 
r ^ ^>^sa^^^^c <t T?*#tt J: ^AftOTW 
r-^^ci^ffl*tc^O^ **ttJ:&Aft£ffi? 

>" #$fts«M -ro 3" *c«a*n**sc. 

v 3 >BJMK** lsX7J*vm&Wrt>tl. 
OTB8ccST^*l*TG>S'*y*© "A ) " teJ:0' 

[ o 104] s"*- y a ) so 



tfHIH^ -9 1112 
r < ^ "7" U ^t* > tc . i-'A y-JU<O^A Y !lWl3i&^. 

^u. ^•m^i^tcA^^f^7D>^^ejs^^ 
[ o 1 o 5 ] f + y 3fO> b > 

[o 107] c<o&mcmn>x< m^m^i zim 
£>> =r << rm^ikm^ i 2^m^jv> ^ > rif^cc^ 

f*ASA' &*^K./. B ! 7 cc^r J: ^ vpja<^> j: 
^*CA**©*ISM^fl>««lft*5iUJa3»6, B8^> 

i *c . vsffl i 3 iinam^^st?ab & r -f y - 

[ 0 1 0 8 3 <.^-(DWPf?IJ> iXcc*Si5Sra^.*^fc» 
~<Z>MffM^fr» a Cj2t^^ # B8fiDV+y* fc O " If 

AadH^ , . ^fttaR' , ~mm~ > 

55^^ 1 0 ©di 1 4 K 4oH^ ^> ffi Tj > r ^7^1 

[0 1 1 0 ] COD**, m^'vlyi- 1 3 1©^^>IE 
t&SPi^ 1 0©*^B* "P42' *-CflSP»Ofctt»* 

^" ?:^ST^^tC, ^lS7 ^>i- 1 3 !1***1S*!BU 

[0111] :XlCtttZ-75>1' !3urr 08'* ^ig 

$n^ e C©-tf7'g^ii[5j|ilcc^!aiiPiJ b r i 0'" ccj: 
-?t. t7@«^j[p43l l, P4312. P4 
3 l 3] tCMB^n-So 
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[0112] en&wrraw^w©**, 

" p 4 3 i i * tew+cmmztix 7? >iEtt»*is i o 

-rf^iTiiartien 2*wxti]£t a i*. i7«a 
^-•7)1 1 2 z<m^c^^xx^^A rmmfflKs 

-F m3^ ^*^cc«Sl/rig«sn*o 

[0113] *um#^^8Smc[>*^6mg 

[0 1 1 4 ] Ctitcjrrt. JMl 8<0&5lcmW±.&3# 

- t>3*s a e - a % wf^&77 $ c t cc v * . 
[oi i5] im&hfr*mmM<D®<£. ffi/ssstcc 

*k 6 ^ xt>fr 0 tc< < Jgmc < u hWiitthti 
[oi i6] ^-*^~*©tii^*#*t*j: 

[0 1 17] **tS^iBi*COJ:^^TOtC^«5a^ 
*«gtc jmi*. ^fSsp i 3 7w rii^rx^ 

[01 183 Haoos^y* *'D ) - cc^gcc 

<D*><o&miR2 Ghtcmc * -r < 7$m±!&& i 2 1 
xr& ^ r ^ r tmmn-r - 1 2 2 <o s - f - m 0 
4" *5giR£*v *-0>$*3i. @l 8<Ecfc^cci 
l/Ttt&ffiiliSoc^ < Til 

[0119] isfruu&h. Httotssr 



!13) ^¥9-9 1112 

aadMSw: l 0 f£*C;fc ^ -rf < 7il^£s£S|i l 2 
l^CKRdn^^^-rTtiaiilR^- Ft* fc m0 5~ 

[012 o ] *•■•< ur, *^sccj:n^rv 

[0121] W^, M^^BSOt^- 'J* ^G> " <7>i§ 
*i»^^^±*V^B^#^^?c^ 77 ij^-i 

[0 122] C<Di%iZKL&~?% <7>£ *)ICU h 0 ^. ±IB 

20 KKJ:&#£^£*&?Tbtv ^!§77>tl 3 l cc-^ ^ 
>l5iSBUliiIj 1 0<?>J: 0^1f^@^ 'p 4 3 1 1 1 " « 

[0123] c coeioms. femi&mm&M&s z 4 # 

v TMSftCff 1 2 ^^ftiiT^^Sai - P 4 3 1 1 

1" . trut>% 'MAfflc^b^ motl^o 
>431 1 r fl*Bk«Wir, wm-p y 1 3 1 
\tmfi\(D ^>t*-$ mE<C>—r>X$> Z p rolot^ 
30 > ^ ^ y > ^ ^1i<7>^M^M^:?T *7 • 

[0124] T^^fe, "p 4 3 1 1 1 " <?>±ttrf-^ 

st^mi/, -p 4 3 1 r* ic&mmtfsmmmbL, 

*fifeUA:^5s5§Ri! fc r 1 2' ^Kcaffiprte^^iSaBU 
feWB^'J^^ <-7i*tj;^tc&>, ^p4 3 1 r ^.^Bfe 

"p43i r oite^-.'ur-^-s - p 4 3 i ~ <p 

[ 0 1 2 5 ] HKCO w p 4 3 1 - &HtR(C:&R'r*ft: 
40 'P4 3" © ^ti?^^ ^ri^^O, IDIil^C^ 

'j ^ ~r < 7'.m&$ti. m 2 o>mmm - r o 9 - 
&&m$ti. m 1 1 <d£ *>ic+r'z?m&ftW 1 p a 3 0 

1 . p 4 3 0 2 . p 4 3 0 3 ] &&M2ti2>* 
[0126] *tiS7 r ? >tl3ll*-p4 30 r«) 

fc r l 4 - &&mu> VzrmtiK&t [ p 4 3 0 I 1 . p 
4 3 0 1 2, p 4 3 0 1 3 ] ££sTr £ 0 

[0127] wearas^'j 2 r a 7X'$>zwm& b p 

50 4 3 0 1 1 '© • % *A*.*ttW^tt'" CC^?T3&ffI'CM!i 
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[0128] vtcc^^g^ ~p43Ol2"0~ntem 

[ o i 2 9 1 -e or , *©*«t*iaiiB*fri or «w* 

^A^^^^c^tC, :X©^^eg "p430 1 3'O 10 

•?mU7kn [ p 4 3 0 ! 3 1 , p 4 3 0 1 3 2. p 4 3 

0 133] 

[013 o ] ctoh<ov7mni& i i<C'^Tm.ni£tei,>x 

BSCDWy* *G> * <£c£^ftMl§£?T:bn. o 
r*^sti>' ^ < rsiacC'fttt^^Ptsor^ra^io 
Mis fcaisfc-r ^Ci a* m$ t ft &<> 

[0 1 3 I ] HI lit, HI Ua8CPW*M:t *G> - 

otts^orti^, 20 

[0132] <JK=<MWfll> &K*ftifijUI*ft*.fc* 
=©*ff«K:oi»ri*^^ # 1218 0^11* ~D> ' * 
rcC'^ga>»<:. ©S^"*- 1 ;* ~E> - 4>J:*tt:f<J£l 

5 1 oawtoayjofe-* *v Ti»«:»or*u«*w 
•c-^ > r y rfia <t tip* * >• ~r y y% m vv>&t>&& 
$>tc £ *> urns. &&<»-7fr * * - ?>immmc a t > 

pjgferab^fc, o*»o, #£«fl<D^ri*c<M:$w* 30 
H^**rfcr**« 

[0133] C<D£ ^ft*i8u:jfiM0fcii^, $£151* 
£ AW^ r < 7 o>« 0 Sr * <Mt 0 . >' r 

^ 7* m&tm 1 2 j»*Hs©a6n^i*skfcK:y ^ < rm 

[0 134] S8^>t''t ~E> " KfcW£U60>J: 
? >igtfc*P<Z>l*.®lii2| 1 2CC'ct ^ tc. - 1> 4 3 

1 3" <d -i-tJiwr 0ffl4i0>f^rfiug4r®AA 40 
^on^R^^c. c cr**«j^<^A/5^«^ 
<D&1R&mxft < , >*r < 7£S!£K£<-nK#$mH 

[0 1 3 5 3 C«*lfcef$c«-OS&f^«fl>iS^ill1iS 
CCtftR«3*&£± 1/ % - p 4 3 i 3 <Q$V&mX'<hZ> 
" p 4 3 I *" <D -jgftgjjr 7**8*0. @2G>*fIS»9l 

k r 15' ^m^f$n«s. c^is^ilo^tf^r 
>t" i3ii* ^-trs* ( [sir, a 
ahm] > ) - ii^ftffr, ' < rasa-ess 1 2 so 



«BHP9-9 1112 

26 

tcSWft-* 7if^CDi2^^^T^o C Office J: 
tti$7v>r 1 3 i tcsg-r, 

[0136] mz? y >i~ 1 3 i t*C«>5g*KK:f£c\ IS 

o. H^#7SaB*fesfttc^t^r-e<D -€«r > *^ 
. "item" x^wtcf^or^ ? 

^^S ! 14><J:9&ftK [eilJlil^^Ar'^l^fel^ 

^^'r-f o o^WdJttc**Witr3 o»»«T->*ti c 

^6^<ACdtt9tt^r3 0»S^D*-T. IBIlllRU 
C C ^f=>^ <X C 3 0 ^l3:i'i^W 3 0 ^tJi'^l^c 

0, mmmo>mm<om^mnmm 
[0137] or, *^gtcj:n^^#^^-r 

^> AHtfj**?: ffiC » , 5 tc*i\mmv>$ Hii^tC IS 0 r fJ^ 

[0138] <j^3(D«WPf?fl) f*JJ^^^^Jf fe**A^ 
OPI^, IE O < ft ^ &a&«: A -7- A^*|baiA5S(C 

*»0fej:^ja*S'&r'<D«®i4fp««:. ^xtc, ^m>tt 

^^^AA5«ffi^e>0*.*A^«:*«>A:^, *Jffi«^ 
IfeO^^ftifi^, «*©-7^^*-5Tjl,«ISftBfc:fe 

[0139] IrfrUtttfih. C<»£5ZLm&lCi>*m&: 
X'&. mz-y^yr 1 3 1 ^fn«*i©»Sg«>*»«r3S|- 
HaB!!Bt!T«S«>*tt4*it. SttOfct^BfcfciWat? 

[014 o ] ccD«r«#^Atc^t - -7 
^o^x^Ai oro«ii£i^finr c^-^ a 

A&E, 4#^is*fe1ii^^^o^^^<3[>AA?*i"a<^ 
<t c - ^ s.*r li* ^ r t * i>a^ > -?'v?(^^r 

[0141] c^ct^ft^w^xo^^td^r^^n 
tt* 3 o * 5 ? j t c ^ ifefit <om&xw 

7^ HI l^^r^r^i, 
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[0 1 4 2 3 CCX\ KWAA^g^fe 

[0143] C<Dfc&>, >430 l33r 

liSteBl/, fc D430 ! 3 3^ <D 'A^** 

CO - r 1 9^ OTG-rf^VrflS-CRA**** 

3 0 13 3 1" ktm 1 4«cwj: itcwmnm i p 

4 3 0 1 3 3 0 1, p 4 3 0 1 3 3 0 2 ] tC*PJ5<*n, 
[0144] OgO, 

[0145] c^orat^sar*^»ft»K(»^tt 

U. V-^mm P4 3 0 1 3 3^ \mzf9ztk?&. -XtC 20 
jMdn/cMM -r20 r t*, A^^t-iOT^ 
r < 7ilii£^>0t£l,, HAJj$-tt*«S*IBi|-C*0, 
C©*tS5»BI«:J:t) * P 4 3 0 1 3 3" itw^tfsn, 
Wm»n I P 4 3 0 1 3 3 1 1 , p 4 3 0 1 3 3 1 
2, P4 3 0 1 3 3 1 3) KJWaSft*. 
[0146] - P 4 3 0 1 3 1 1 - DMA5B4R 

(A2*&l* <H***tt> ) " tiM^^IsSiel "r 1 
2" 'C^^SS^M [ P 4 3 0 1 3 3 1 1 1 , p 4 3 0 
133 1 12) £JMfl$*i&. 

[0147] It^mm fc p430 !33l 11" © fc M 30 

y&it-T-yn i 3 3oi*^icm*> *v rii^?x^ 
i 2<£^tf£uom>n*. 

[0148] Rseca^feA 5 cc, < 7«3ttC* 1 
2tfH6iC5tr3W* r ./ TlMS***-^ 1 2 2££ 

*fl*S§S0>ig*K£ v ^ £ * * $ Z J: ^ ttttBKff 

[0149] ?ztt>% % 

u. &st* rigs#j & r o a ^ *>a, i; j ±SMrr*. 

t'U^6<^*>< fc<*OJ 50 



nm*9 - 9 1112 

28 

I 0 1 5 0 ] $»< *^Stc J:n^]^#iCC^a 

[0 1 5 1 3 KK, mi^^SS#d«t/<^ofei: 
a too, ^-^A^tcjcO^AS/}^^ 

[0 1 5 2 3 FlJiU^j^A/J^T^ ^ v r 

$ c i - r , mmm*y>z><Q hi) ***-c* w 
iJB©itff«<tlW*c*^i8a*tt«:fl: -7 r ^ < 7 si 
^ < 7SfS©^*^cir^*fe$n^^, -e^iswcc 

fe^ Cii^?: AtfJAtcWc^ J: 5% > r v 7ftJ^iiiM 
[0153] ^ < 1/ r % CCC-J:^ ^m^^g^ £ft 

it. «-w.^«<«ft«i6ot:AH«}*2s:t*R^ 

&M9l«fMDt Aid t X. 9\&®&&m7ktcfcCX&m 

v\&ftiittmRuxmm?z>v>x. mm%±<om* 

[0154] r±\*. ±&<DmX'&s A/5^5 1 1 (TMvfrC 

^*x;*j77J<<<AiEmjHLi i i, A/?*- K&is&n 

[0 15 5] Sfc, ±ife©PIIr'it, A^iffl/5gP^*« 

[0156] ^y-c, ±i&otc&xit. *-r<?mm&:m 

c t x&3iotc&&£ ox<o 
* r < 7*s**aw*ftt < ^ <>-**ta*iBir £ c <t ^ 
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imtt&fi®<ome&&~?<t: *> ^farh C t *> "itfeX 
[0157] c<o£ ^cc. ^m^mt h-mm&zo 

^ r $mo>im ^i2^ c i cc j: o . J*?* 7 %mo> 

[0 159] iatCDW-Cl*, l^g^ >£*i£#5$ 

^§njtfet:'£>£. 30 
[0 160] ±^«'f?l]T'ti, Wffi«^Jfl3nJtew 

[oi6i] i^cc-wr-i*, mm&fiimsmtt 

^6Uri^^ fW!i£K^&i'tt&<oi*Aft:fr&t£ 40 

Bae3n&*>©"ci*<f < . \&t>i$mc^>xhw&^ 

[0 i 62] £;fc mtcffltlix, s r < r&mixiz&u: 

> r -/ r o < itmm^eiw* > ^> r ^r^-^ia^ 



^¥9-9 l l I 2 

30 

[ 0 1 6 3 J GUtf. ±IS^: ^ < 

tc^-r^fyji^isi^^sis^, mux. !c^7-fw 

pFtt(Cft«^B#w^n^«l: i >(Cft^. c^. 

<o*uw#<oea^->f r ^ 7m%xm\\mmt&<o&tov>m 

s=*J>>e>&1fc$ti&o>X\ &MV>&M*\'$&'? iftl/ 
cc. Pfitftfm^Ifi^^E ^ C tip*-*- b'Afsj±cc© 

[0 l 64] «*WflM«>*3^W«r*J:^ft^ 

■=e ^HS^^rtf ^ J: 0 oc U . ffl^#^35«r * * ^ cc 

t>CLi*X'*Z>*: Jt >ictsZ>. 

[0165] 5 6cc>' ^ < 7SlSft«*cc ID3*i a 
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(54) MULTIMODAL INTERACTIVE DEVICE AND INTERACTIVE 
METHOD 

(57)Abstract: 

PROBLEM TO BE SOLVED: To enable 
interaction by dynamically switching 
input/output media in response to 
situations. 

SOLUTION: This device is provided with 
an input means 1 1 for interpreting the 
input data of a user from one device 
corresponding to input medium allocation 
among different devices for various kinds 
of inputs and applying that input 
corresponding to interpretion to a problem 
solving means 15, problem solving means 
1 5 for outputting the result by performing 
processing corresponding to the purpose 
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of a system corresponding to this input, output means 14 for 
interpreting the output of this problem solving means, distributing this 
interpreted result to any device for output corresponding to output 
medium allocation among different devices for various kinds of outputs 
and outputting the result to the user, and medium allocating means 1 2 
for dynamically generating the method for selecting input data and 
interpreting them and the method for combining those data 
corresponding to the condition of interaction with the user by the input 
means 1 1 and the output means 14 in order to construct the input data 
to the problem solving means from the input data applied from the user 
to the plural input devices. 
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CLAIMS 
[Claim(s)] 

[Claim 1] In the dialogue equipment which performs necessary 
processing by the dialogue with a user, and outputs a result An input 
means to interpret the input from at least one device for an input 
according to input media allocation among the various devices for an 
input with which it differs for acquiring the information from a user, and 
to generate response information based on the interpretation, A 
problem-solving means to carry out processing according to the 
necessary object to each of this generated response information, and to 
output a result to it, An output means to output the result of having 
been based on the output of said problem-solving means, from the 
device for an output which followed output media allocation among 
various different devices for an output, A means to choose said 
response information given from said input means to said problem- 
solving means from said response information generated for said every 
input media, Said input media allocation which assigns at least one of 
the means which combine the response information generated for said 
every input media, And said output media allocation which assigns at 
least one of said problem-solving means to the means which combine 
the means and them which choose the information from this problem- 
solving means in order to acquire the print-out outputted to an output 
means, Multi-modal dialogue equipment characterized by providing a 
media allocation means for it to be alike, and for it to be based and to 
perform input or output media allocation according to a dialogue 
situation with said user. 

[Claim 2] In the dialogue equipment which performs necessary 
processing by the dialogue with a user, and outputs a result An input 
means to interpret the input from at least one device for an input 
according to I/O media allocation among the various devices for an input 
with which it differs for acquiring the information from a user, and to 
generate response information based on the interpretation, A problem- 
solving means to carry out processing according to the necessary object 
to each of this generated response information, and to output a result to 
it, An output means to output the result of having been based on the 
output of said problem-solving means, from the device for an output 
which followed I/O media allocation among various different devices for 
an output, A means to choose said response information given from said 
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input means to said problem-solving means from said response 
information generated for said every input media, At least one of the 
means which combine the response information generated for said every 
input media And at least one of said problem-solving means to the 
means which combine the means and them which choose the 
information from this problem-solving means in order to acquire the 
print-out outputted to an output means Multi-modal dialogue equipment 
characterized by providing the media allocation means which assigns I/O 
media according to a dialogue situation with said user based on the I/O 
media allocation which ****** is made to correspond and is assigned. 
[Claim 3] In the dialogue equipment which performs necessary 
processing by the dialogue with a user, and outputs a result An input 
means to interpret the input from at least one device for an input 
according to I/O media allocation among the various devices for an input 
with which it differs for acquiring the information from a user, and to 
generate response information based on the interpretation, A problem- 
solving means to carry out processing according to the necessary object 
to each of this generated response information, and to output a result to 
it, An output means to output the result of having been based on the 
output of said problem-solving means, from the device for an output 
which followed I/O media allocation among various different devices for 
an output, The dialogue management tool which directs modification 
processing of I/O media allocation according to the processing result 
from said problem-solving means, and manages the I/O by said input 
means and output means, In order to obtain the input from the input 
data given from the user to at least one input device to said problem- 
solving means In order to obtain the expression directed to a user from 
the output of the at least one and said problem-solving means of the 
combination of the input approach of of the device and input data which 
are used among said devices for an input according to a predetermined 
regulation While carrying out I/O media allocation for making the both 
sides of at least 1 ** of the combination of the output method of a 
device and output data which use among said devices for an output 
according to a predetermined regulation correspond, choosing, and 
assigning It is multi-modal dialogue equipment characterized by providing 
the media allocation means which processes the modification in order to 
direct modification processing of I/O media allocation, and carries out 
new media allocation at the time of a carrier beam. 

[Claim 4] A problem-solving means to carry out computation according 
to the object of a system to an input response, and to output a result to 
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it, An input means to interpret a user's input data from at least one 
device among various different devices for an input corresponding to I/O 
media allocation, and to give the input corresponding to an interpretation 
to said problem-solving means, An output means to obtain the 
processing result from said problem-solving means, to distribute this to 
the device for an output corresponding to I/O media allocation among 
various different devices for an output, and to output to a user, The 
dialogue management tool which directs modification processing of I/O 
media allocation according to the processing result from said problem- 
solving means, and manages the I/O by said input means and output 
means, A means to choose the input given to said problem-solving 
means from said response information generated for every input media, 
And at least one of the means which combine the response information 
generated for every input media And in order to generate the expression 
directed to a user from the output of said problem-solving means, while 
choosing said I/O media allocation which is the combination of the both 
sides of at least 1** of the means which combines the means and them 
which choose the output It is multi-modal dialogue equipment 
characterized by having a media allocation means to process the 
modification in order to direct modification processing of I/O media 
allocation, and to choose new I/O media allocation, at the time of a 
carrier beam. 

[Claim 5] In said dialogue management tool, one [ at least ] situation of 
an input unit and an output means of operation is supervised suitably. 
When the failure over utilization of the I/O media allocation which 
current selection is made and is used occurs, while having the media 
monitoring function which notifies the fault information showing 
generating of this failure to a media assignment-decision means Claims 
2 or 3 characterized by having the function to change I/O media 
allocation suitably if this fault information is received in said media 
allocation means, or multi-modal dialogue equipment given in 4 any 1 
terms. 

[Claim 6] When information required for said problem-solving means for 
the problem solving of a system needs to be acquired from a user the 
input request function for which it asks so that input request information 
may be generated to said dialogue management tool and I may have a 
user input said input required information — or Information required for 
said dialogue management tool since the output to a user is constituted 
according to the I/O media allocation to which said output means is set 
the output request function for which it asks so that the output request 
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information which asks for an output may be generated for a problem- 
solving means and a problem-solving means may output said output 
required information to it, and **, while having at least one function 
inside For said media allocation means, directions of I/O media 
allocation modification processing at the time of a carrier beam Claim 3 
characterized by having the function to perform new media allocation 
using at least one of said input request information or said output 
request information, or multi-modal dialogue equipment given in 4 any 1 
terms. 

[Claim 7] Interpret the information acquired from the user by the 
dialogue with a user, and the information corresponding to an input is 
generated. The information corresponding to an input is sent to the 
problem-solving equipment which carries out processing according to 
the necessary object to each of this information corresponding to the 
generated input, and outputs a result to it through means of 
communications. In the dialogue equipment which outputs the processing 
result of this information corresponding to an input acquired from 
problem-solving equipment through means of communications by the 
dialogue with a user An input means to interpret the input from at least 
one device for an input according to I/O media allocation among the 
various devices for an input with which it differs for acquiring the 
information from a user, and to generate response information based on 
the interpretation, An output means to output the result obtained from 
said problem-solving equipment through means of communications from 
the device for an output which followed I/O media allocation among 
various different devices for an output, Said sending corresponding to an 
input generated in the input means to said problem-solving equipment, 
and the means of communications which performs the receipt of the 
problem-solving result which problem-solving equipment generated and 
was outputted, A means to choose said response information sent to 
said problem-solving equipment from said input means from said 
response information generated for said every input media, At least one 
of the means which combine the response information generated for said 
every input media And at least one of the means which combine the 
means and them which choose the information from this problem-solving 
equipment in order to acquire the print-out outputted to an output 
means from said problem-solving equipment Multi-modal dialogue 
equipment characterized by providing a means for the media allocation 
which performs I/O media allocation for generating the I/O media 
allocation which is the combination of ****** according to a dialogue 
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situation with said user, and performing a dialogue with a user. 
[Claim 8] Interpret the information acquired from the user by the 
dialogue with a user, and the information corresponding to an input is 
generated. The information corresponding to an input is sent to the 
problem-solving equipment which carries out processing according to 
the necessary object to each of this information corresponding to the 
generated input, and outputs a result to it. In the dialogue equipment 
which outputs the processing result of this information corresponding to 
an input acquired from problem-solving equipment by the dialogue with a 
user An input means to interpret the input from at least one device for 
an input according to I/O media allocation among the various devices for 
an input with which it differs for acquiring the information from a user, 
and to generate response information based on the interpretation, An 
output means to output the result obtained from said problem-solving 
equipment through means of communications from the device for an 
output which followed I/O media allocation among various different 
devices for an output, The dialogue management tool which directs 
modification processing of input media allocation and output media 
allocation according to the processing result from said problem-solving 
equipment, and manages the I/O by said input means and output means, 
Said sending corresponding to an input generated in the input means to 
said problem-solving equipment, and the means of communications 
which performs reception of the problem-solving result which problem- 
solving equipment generated and was outputted, In order to acquire the 
information corresponding to the input given to said problem-solving 
equipment from the input data given from the user to at least one input 
device In order to obtain the expression shown to a user from the result 
obtained from at least one and said problem-solving equipment of the 
combination of the input approach of of the device and input data which 
are used among said devices for an input according to a predetermined 
regulation While carrying out I/O media allocation for making the both 
sides of at least 1** of the combination of the output method of a 
device and output data which use among said devices for an output 
according to a predetermined regulation correspond, choosing, and 
assigning It is multi-modal dialogue equipment characterized by providing 
the media allocation means which processes modification processing 
directions of I/O media allocation in order to make the change at the 
time of a carrier beam, and carries out new media allocation. 
[Claim 9] While recording the I/O media allocation which has the media 
allocation storage slot which records I/O media allocation in said media 
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allocation means, and is chosen as said media allocation storage slot 
now Directions of I/O media allocation modification processing at the 
time of a carrier beam Claims 2 or 3 characterized by having the 
function to perform new media allocation using the I/O media allocation 
information currently recorded on said media allocation storage slot, 4 or 
7, or multi-modal dialogue equipment given in 8 any 1 terms. 
[Claim 10] It has the media allocation history storage slot which 
memorizes the sequence of the I/O media allocation chosen in the past 
in said media allocation means. When the sequence of the I/O media 
allocation chosen by then is memorized into said media allocation history 
storage slot and I/O media allocation is changed into it While adding the 
I/O media allocation then chosen to said media allocation history 
storage slot, directions of I/O media allocation modification processing 
at the time of a carrier beam Claims 2 or 3 characterized by having the 
function to perform new media allocation using the I/O media allocation 
information on the past currently recorded on said media allocation 
history storage slot, 4 or 7, or multi-modal dialogue equipment given in 8 
any 1 terms. 

[Claim 11] In said dialogue management tool, when said problem-solving 
equipment needs to acquire information required for a ** sake from a 
user to the problem solving of a system As opposed to the input request 
information which problem-solving equipment was generated and was 
sent through means of communications to the dialogue management tool 
the input request receptionist function for which said input means is 
asked so that I may have a user input said input required information — 
or Information required for a dialogue management tool since the output 
to a user is constituted according to the I/O media allocation to which 
said output means is set the output request function for which it asks 
so that the output request information which asks problem-solving 
equipment for an output may be generated, it may send through said 
means of communications and problem-solving equipment may send said 
output required information through means of communications, and **, 
while having at least one function inside It is multi-modal dialogue 
equipment according to claim 8 characterized by equipping said media 
allocation means with the function to perform new media allocation for 
directions of I/O media allocation modification processing using at least 
one of said input request information or said output request information 
at the time of a carrier beam. 

[Claim 1 2] It is a deed about the input step which the input from at least 
one device for an input interprets [ step ] according to I/O media 
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allocation among the various devices for an input with which it differs for 
acquiring the information from a user in the dialogue approach which 
processes business by the dialogue with a user and outputs a result, and 
generates that information corresponding to an interpretation, and the 
processing according to the necessary object to this information 
response that generated. The output step which outputs the result of 
having been based on the output of the problem-solving step which 
outputs a result, and said problem-solving step, from the device for an 
output corresponding to I/O media allocation among various different 
devices for an output, The dialogue management step which directs 
modification processing of I/O media allocation according to the 
processing result from said problem-solving step, and manages the I/O 
by said input step and output step, In order to obtain the input from the 
input data given to at least one input device among said two or more 
input devices to said problem-solving step At least one of the 
combination of the input which chooses the device used among said 
devices for an input, and is generated for every input media In order to 
generate the expression shown to a user from the output of said 
problem-solving means And the inside of said device for an output, While 
choosing said I/O media allocation which is the combination of the both 
sides of at least 1 ** of the combination of the output method of a 
device and output data which use according to a predetermined 
regulation and carrying out this input media allocation It is multi-modal 
dialogue equipment characterized by having the media allocation step 
which processes the modification in order to direct modification 
processing of these I/O media allocation, and generates new media 
allocation at the time of a carrier beam. 



[Translation done.] 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the human machine 
dialogue equipment which combines and has two or more available I/O 
means, and its control approach in an information processor. 
[0002] 

[Description of the Prior Art] A computer in recent years A keyboard, a 
mouse, a microphone, a touch panel, Two or more input means by which 
the classes of an image scanner, a video camera, a pen, a data glove, 
personal sensor, etc. differ, Namely, an input device can be equipped and 
various information can be inputted now. Various information also to 
output devices, such as a display and a loudspeaker, moreover, a spoken 
language and a sound effect, The function which can be outputted with 
various gestalten of music is equipped, these various I/O devices are 
used effectively, it depends for them, and the demand of a human 
machine interface which is easy to use is increasing. Moreover, using 
these I/O devices, while a user directs the object on a display with a 
finger, a command is uttered with voice in recent years, Equip the input 
means in which the complicated input which combined two or more input 
devices is possible, i.e., input media, and two or more output devices, 
such as a display and a loudspeaker, are used. For example, the 
expression of human being's face displayed on the display while telling 
the message in the spoken language from the loudspeaker, By equipping 
the output means in which the complicated output which combined an 
output device, and two or more of its contents of an output and formats 
with condition of telling nuance by the sound effect from a loudspeaker 
is possible, i.e., output media Development of the multi-modal interface 
technique in which the ease of using of an interface, the quality of signal 
transduction, and effectiveness are raised is performed briskly. 
[0003] By the way, in order to realize transfer of the information 
between smooth and conventionally natural users and the application 
software on a computer, it considered that transfer of this information 
was a dialogue between a user and application, the dialogue plan was 
developed based on the dialogue regulation recorded beforehand, and it 
was carrying out realizing a multi-modal dialogue in the combination of 
the I/O approach decided in the dialogue regulation etc. 
[0004] 

[Problem(s) to be Solved by the Invention] However, the combination of 
the utilization method of the device and device which are used for I/O in 
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the phase of each dialogue, namely, by the method beforehand described 
in the dialogue regulation, media allocation (media location) For example, 
a situation for which it is not appropriate that circumference noise 
sound volume uses voice as an I/O means greatly cannot be coped with, 
Media allocation is fixed with the flow of a dialogue, the combination of 
an I/O means was chosen according to the situation, and there was a 
problem that it could not converse with a user. 

[0005] For example, the case where the dialogue equipment which can 
be inputted by the user with voice is used is considered. In this case, 
with a current technique, since the technique of speech recognition has 
not matured enough, even if some users generate the same word 
repeatedly, that recognition goes wrong may occur. Generating many 
this, when a peculiarity is in pronunciation too much, the probability of a 
recognition success by the dialogue equipment side becomes very low. 
[0006] Nevertheless, conventional multi-modal dialogue equipment had 
the problem of the futility of time amount having arisen or making a user 
sensing stress in order to require such a user repeatedly to recarry out 
an input with voice. 

[0007] Moreover, although having considered the itinerary system by 
voice directions of the configuration of searching the tourist resort of 
relevance from a database and outputting a retrieval result to an output 
means by speech recognition, for example to a name of a place input 
with voice this itinerary system asked the user for the name of a place 
input with voice A user On the other hand, since [ for example, ] the 
name of a place of "Ryoan-ji" (************) was changed between 
"************" and reading and was uttered, Since it did not have a 
means to perform I/O media allocation dynamically in conventional 
multi-modal dialogue equipment when a system fails in retrieval, it was 
difficult to recover communication with a user. 

[0008] That is, although that a user does not know reading of the right 
proper noun " ************" has led to failure of communique SHOKU 
with a user If this is also changed to the format which makes input 
selection with the kanji, although possibility that communication 
recovery can be performed will be size very much Since it was not able 
to say that such I/O media allocation was changed according to a 
situation, it will lapse into a stiffness condition and it will be said that 
advice service cannot be performed as a result. 
[0009] Moreover, the output gestalt is being fixed to the gestalt 
programmed beforehand when showing a user the content of the 
database retrieval result in a certain dialogue equipment, for example 
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from the former. That is, although the same output gestalt [ be / no 
relation in the output number of cases of a retrieval result ] is taken, for 
example, there is a retrieval result of dozens of affairs or hundreds of 
affairs, there were quite a few cases where it became the I/O method 
which is hard to use unintelligibly for a user of being as displaying the 
retrieval result which has only several affairs conversely by the tabular 
format **** [, and ]. [ reading them out with voice one by one ] 
[0010] Furthermore, since a system was not able to perform a dialogue 
with a user to compensate for the media allocation which a user 
demands, the user of a system also had the problem that a system had 
to output and input according to the media allocation opted for and 
specified. 

[001 1] As for the I/O means which is easy to use for condition that 
another user likes carrying out the keyboard entry of the run string, by 
the user, differing is common, although it is not necessarily carrying out 
[ everybody ], for example, as for the input means which is easy to use 
for a user although there are various things in an input means, a certain 
user likes an input with voice. 

[0012] However, the difference among these individuals was not taken 
into consideration in conventional dialogue equipment, but the 
combination of a fixed I/O means was offered, an I/O means cannot be 
selectively used now by liking of a user, and there was a problem that 
user-friendliness could not say it as a good thing. 

[0013] Moreover, in conventional multi-modal dialogue equipment with 
two or more input means, there was a problem of it having been difficult 
which input means to be used or for a user to recognize ** whether it is 
available when there is which input means, and giving a user puzzlement 
and stress on the occasion of an input. 

[0014] Furthermore, there was a problem that where of using voice for 
an input or an output even when it is not desirable to use voice — the 
external noise becomes intense — etc. cannot change the I/O method 
according to change of an external-environment-element in the 
conventional dialogue equipment which cannot perform media allocation 
(media allocation) dynamically, for example in the dialogue equipment in 
which a conventional input or a conventional output with voice is 
possible. 

[0015] Thus, the user made it force with conventional multi-modal 
dialogue equipment to perform a system and a dialogue according to the 
combination of the I/O mode beforehand decided under the dialogue 
regulation which a system has, since it was not the interface which is 
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easy to use intelligibly for a user, on the occasion of the informational 
input, stress was sensed, and it is harder for a user to understand an 
output and, occasionally he had become also with the cause of failure in 
I/O. 

[0016] Moreover, in voice word recognition mode, as it said that there 
might be the input, the failure, and the incorrect input resulting from 
recognition failure, the object of a dialogue had been un-attaining as a 
result also in the case of failure [ in the I/O resulting from the property 
of the specific selected I/O means ]. 

[0017] Then, the place made into the object of this invention is to offer 
the multi-modal dialogue equipment which can serve as an input/output 
interface which a user tends to use. It is in creating the multi-modal 
dialogue equipment which can offer the input/output interface which is 
easy to use for a user by having the function which chooses dynamically 
media allocation of I/O especially suitable at a certain event according 
to various situations as a front end of an application program in this 
invention. 
[0018] 

[Means for Solving the Problem] The multi-modal dialogue equipment 
concerning this invention A means to generate dynamically the approach 
of combining the approaches and those data for choosing and 
interpreting input data based on the rule defined beforehand in order to 
build the input data from the input data given to two or more input 
devices to [ from a user ] an application program, A means to generate 
dynamically the approach of choosing output data in order to generate 
the expression shown to a user from the output data of an application 
program, and the approach of the combination and an expression of 
those data based on the rule defined beforehand is provided. It is 
characterized by the ability to choose the combination of a means 
suitable about each of I/O at each time of a dialogue with a user 
dynamically, and continue a dialogue. 

[0019] In case a user performs a computing system and a dialogue 
according to this invention, it becomes possible to choose and use the 
method of I/O using the I/O means which chooses the I/O method 
reflecting an available I/O means at each [ of a dialogue ] event, and a 
user tends to use for the user. 

[0020] Moreover, according to this invention, media allocation (media 
allocation) is fixed with the flow of a dialogue, chooses the combination 
of an I/O means according to a situation, and becomes possible [ solving 
the problem of the conventional multi-modal dialogue equipment that it 
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cannot converse with a user ]. 

[0021] Furthermore, according to this invention, if it is conventional 
multi-modal dialogue equipment, a specific user will be repeatedly 
required to recarry out an input with voice, [ when there is a problem of 
the futility of time amount arising or making a user sensing stress ] It 
becomes possible to use a computing system, without solving this 
problem and a user sensing the futility and stress of time amount by 
providing the means which chooses and changes an I/O means 
dynamically. 

[0022] Furthermore, according to this invention, when interpretation 
processing of an input goes wrong, by providing the means which 
chooses and changes an I/O means dynamically to having been difficult 
to recover communication with a user in conventional multi-modal 
dialogue equipment, this problem can be solved and handicap 
NYUKESHON with a user can be recovered promptly. 
[0023] Furthermore, if the gestalt of a voice output is adopted in order 
to tell few retrieval results when showing a user the content of the 
database retrieval result in a certain dialogue equipment, for example 
from the former A retrieval result reads out dozens of outputs of the 
retrieval result with voice, even if there are hundreds of affairs, In order 
to take the same output gestalt [ be / no relation in the output number 
of cases ] and to make easy treatment of the retrieval result of a large 
quantity at reverse, when the tabular format is adopted, even if it is as a 
result of [ only with several affairs ] retrieval, it is [ fixed ] as displaying 
by the tabular format ****. 

[0024] However, by establishing the means which chooses and changes 
an I/O means dynamically in this invention, and choosing and changing 
an I/O means dynamically, when becoming such an I/O method it is hard 
for a user to use unintelligibly, this problem can be solved and an I/O 
method intelligible for a user can be taken now. 

[0025] Furthermore, according to this invention, the difference in the 
individual about the I/O means which is easy to use in conventional 
dialogue equipment is not taken into consideration, but it becomes 
possible to use the combination of the I/O approach which each user 
tends to use for each to the problem that the combination of a fixed I/O 
means was chosen. 

[0026] Furthermore, according to this invention, it sets to conventional 
multi-modal dialogue equipment with two or more input means. As 
opposed to the problem of it being difficult for a user to recognize 
whether it being available when there are which input means being used 
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and which input means, and giving a user puzzlement and stress on the 
occasion of an input By providing a means to display or put up the 
means which can be used for an input by the user, this problem is 
solved, a user is enabled to recognize an input means easily, and it 
becomes possible to offer the interface which is easy to use intelligibly 
for a user. 

[0027] In this way, although a user is dependent on the quality of the 
information which a user and an application program search for, an 
amount, and an I/O means with the available computing system to be 
used, it becomes possible to use the human machine interface 
independent of an application program. 
[0028] 

[Embodiment of the Invention] This invention is providing a user with the 
I/O approach according to a situation by providing a means choosing the 
combination of a means dynamically, about each of I/O. By providing a 
means to realize the interface which is easy to use and to make a user 
know an available input means A user is provided with a more intelligible 
input environment, the combination of an I/O means is changed further, 
failure in a dialogue with a user is recovered, the multi-modal dialogue 
equipment which can continue a dialogue is offered, and the detail of 
this invention is hereafter explained with reference to a drawing. 
[0029] Dr awing 1 is the block diagram of the multi-modal dialogue 
equipment concerning the example of this invention. As shown in 
drawing 1 , the multi-modal dialogue equipment 10 of this invention 
consists of a total of four components, the input section 1 1 , the media 
assignment-decision section 12, the dialogue Management Department 
13, and the output section 14, performs interactive processing with a 
user through the physical device connected to a system, and performs 
intermediation of the problem-solving means (background application) 1 5 
with a user. 

[0030] The input section 1 1 is for incorporating the output of various 
kinds of input device IDs Va, such as a mouse device, a microphone, a 
television camera, a keyboard, a tablet, an image scanner, and a sensor, 
- IDVn, and the output section 1 4 generates the actuation output to 
various kinds of output device ODVa-ODVn, such as a display, a 
loudspeaker, a lamp, a printer, and a plotter. 

[0031] The media assignment-decision section 12 is equipment for 
determining the combination of the utilization method of the various 
devices for the output for an input, and these various devices, i.e., media 
allocation, (media allocation), and has the function which can choose the 
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best combination which sometimes comes out. 

[0032] Moreover, the dialogue Management Department 1 3 controls the 
approach of the dialogue between a user and a system, such as requiring 
media allocation or generating the plan of a dialogue based on the 
information delivered and received between the problem-solving means 
1 5, the status information from the input section 1 1 and the output 
section 1 4, etc., and flow. 

[0033] The problem-solving means 1 5 is the application which is the 
application which performs the reply and answerback to an input, and 
this issues the reply of the object by the dialogue, or does work of the 
object, for example, is good anything. [ of a sightseeing advice system, a 
commuter pass ticket sale machine, etc. ] A block diagram shows the 
detail configuration of the multi-modal dialogue equipment 10 of this 
invention to drawing 15 . 

[0034] (Detail of the input section 11 and the output section 14) As 
shown in drawing 15 , the input section 11 of the multi-modal dialogue 
equipment 10 of this invention consists of three modules, the input 
device processing section 111, the input mode processing section 1 12, 
and the input media processing section 113, and the output section 14 
consists of three modules, the output media processing section 141, the 
output mode processing section 1 42, and the output device processing 
section 143. 

[0035] Here, with the mode, the input (the same is said of output) from 
one device should be interpreted and abstracted. For example, two or 
more interpretations and utilization by the mode called directions for 
actuation and issuance of a command with the mode components which 
interpret it are possible for the data inputted from the mouse device. 
[0036] similarly, media have pointed out what can be passed to a 
referent by making into the content of directions the processing result 
which should combine the interpretation result from the mode of 
plurality (even — also coming out — good), and is obtained combining an 
input. 

[0037] For example, mouse object directions-voice command media The 
input from the mouse device which are pointing devices, such as a 
personal computer, The voice command mode which can interpret a 
command with voice collectively and can interpret the voice which 
carries out command issuance and directs the command to application, 
The interpretation result in the mouse object directions mode in which 
the input from a mouse device is interpreted as actuation of directing 
the object for command actuation is considered as an input. They are 
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the media which perform telling processing "the command directed with 
voice to the object directed with the mouse is published" combining 
those inputs to application. 

[0038] The input device processing section 1 1 1 which is the component 
of the <input section 1 1> input section 1 1 considers the data 
transmitted from each physical input device as an input, and has in a 
component the input device processing components which perform the 
minimum processing for interpreting the data, such as conversion to 
digital data. Device processing components are prepared for every input 
device, and operate to independent and juxtaposition. After each device 
processing component performs the above-mentioned processing to the 
input to a corresponding device, it is sent to one or more corresponding 
input mode processing sections 112. For example, microphone device 
processing components incorporate the voice which the user inputted 
from the microphone with a background noise, for example, change it 
into a digital audio signal with a sample frequency [ of 1 2kHz ], and a 
quantifying bit number of 16 bits, and output the signal to the input 
mode processing section 112. 

[0039] The input mode processing section 112 has the components 
which process the input to each abstract input mode in a component. 
One input mode processing component corresponds to one abstract 
input mode, and abstracts and interprets the data which were sent from 
the one input device processing section and which were operated 
orthopedically for every abstract mode. Each input mode component has 
the response relation of 1 to 1 or one-pair **, as shown in input device 
components and a table 1 . 

[0040] The result interpreted in each input mode processing section 1 1 2 
is sent to the input media processing section 1 1 3. For example, voice 
command input mode processing components search for feature- 
parameter time series by the frequency analysis according the sent 
digital audio signal to a fast Fourier transform, perform speech 
recognition based on methods, such as a compound similarity method 
and a hidden Markov model, further, for example, obtain the highest 
vocabulary of likelihood out of the given command vocabulary for 
recognition, and output the information to the input media processing 
section 113. 

[0041] The input media processing section 113 generates the input 
integrated by combining the interpretation result for every mode of the 
input of a user obtained from the input mode processing section 112 per 
each abstract media, it chooses the input from required media in from 
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among two or more obtained media inputs, following directions of the 
media assignment-decision section 12, and gives the input command and 
the data to the problem-solving means 15. 

[0042] The component of the input media processing section 1 13 is the 
media selecting switch SW which specifies which is sent to the problem- 
solving means 1 5 among the processing results of those components as 
input media processing components 1 1 3a prepared for every abstract 
media. Each input media processing components 1 1 3a receives the 
output data (interpretation result in abstract mode) from one or more 
input mode components as an input, unifies them, and sends the 
interpretation result as media to the media selecting switch SW. 
[0043] The media selecting switch SW chooses and sends the media 
output sent to background application according to directions of the 
dialogue Management Department 13. For example, mouse object 
directions-voice command media input-process components consider 
the output of mouse object directions mode input-process components, 
and the output of voice command mode input-process components as 
an input, and generate and output the interpretation as media combining 
them. And for example, when mouse object directions-voice command 
media are chosen as current input media, the media selecting switch SW 
hands over only the interpretation result of the input by these media for 
the problem-solving means 15. 

[0044] The output media processing section 143 which is the component 
of the <output section 1 4> output section 1 4 chooses the output data 
from the problem-solving means 1 5, following the allocation information 
on the media which the dialogue Management Department 13 gives 
(allocation information), they distribute to each abstract output media, 
and it processes them for every media, and delivers the data which 
divide and unify them per output mode and the output mode deals with 
on each mode processing components of the output mode processing 
section. In order to perform such processing, the output media 
processing section 143 consists of three modules, output media 
distribution switch 143a, output media processing section 143b, and 
media mode interface 143c. Following directions of the dialogue 
Management Department 13 in the output data which background 
application has sent, media distribution switch 1 43a chooses the data 
outputted actually (to user), and distributes it to abstract media 
processing components. 

[0045] The output media processing components 143 have the output 
media processing components corresponding to each abstract output 
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media as a component, and process and send out the output data which 
were able to be distributed to the media in their duty by media 
clistribution switch 1 43a to the form which can be outputted to the 
media. Media mode interface 143c has the media mode interface 
components corresponding to each abstract output mode 1 to 1 in a 
component. Each media mode interface component unifies the output 
from one or more abstract media processing components which the 
mode takes charge of, and delivers data to corresponding output mode 
processing components. 

[0046] The output mode processing section 142 has in a component 
mode device interface 142b which unifies output mode processing 
components 142a and the output data for every mode which process the 
output to each abstract output mode for every output device. One 
output mode processing component corresponds to one abstract output 
mode, processes every output mode to the media data sent from the 
media mode interface components corresponding to the mode, and 
delivers it to mode device interface 142b. 

[0047] One component of mode device interface 1 42b is equivalent to 
one output device component, the output from one or more abstract 
device processing components which the device takes charge of is 
unified, and data are delivered to corresponding output device 
processing components. The output device processing section 141 has 
as a component the output device processing components prepared for 
every output device. Each output device processing component drives 
the output device which takes charge of the output data of a 
corresponding media device interface as an input command and data. 
[0048] (The detail of the media assignment-decision section 1 2) As 
shown in drawing 15 , the media assignment-decision section 1 2 of the 
multi-modal dialogue equipment 10 of this invention consists of three 
storage components of the mode-media allocation conversion table 
attaching and showing the present media allocation storage slot and the 
sign 124 which attach and show the media [ degree ] assignment- 
preference table and the **** sign 1 23 which attach and indicate a sign 
122 to be a module called the media allocation generation section, as a 
sign 121 was attached and shown. Among these, it combines (media 
allocation), and the media allocation generation section 121 generates 
the suitable combination and expression approach of Perilla frutescens 
(L.) Britton var. crispa (Thunb.) Decne. as the data actually inputted into 
application, and its data actually given to a user with reference to the 
media [ degree ] assignment preference table 1 1 2 at the question of the 
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dialogue Management Department 13 according to doubling, and 
transmits it to the dialogue Management Department 13. 
[0049] As shown in drawing 15 , (Detail of the dialogue Management 
Department 1 3) The dialogue Management Department 1 3 of the multi- 
modal dialogue equipment 10 of this invention The condition of the I/O 
mode obtained from the input media processing section 1 1 3 and the 
output media processing section 143, The condition of the data 
exchange with the problem-solving means (background application) 15 
The plan of the dialogue [ by the description information common to two 
or more users for the input/output request from advice or the problem- 
solving means 1 5 of the condition of the problem-solving means 1 5 to 
the problem-solving means 1 5 and the information about the control 
approach of the problem-solving means 1 5 ] based on reception and 
those information as an event is generated. 

[0050] And the approach and flow of the dialogue between a user and a 
system are controlled by directing for the input media processing 
section 113, the output media processing section 143, and the problem- 
solving means 15 in the activation process of the plan. 
[0051] If it considers that transfer of the information between other 
modules, such as the input media processing section 113, the output 
media processing section 143, the problem-solving means (background 
application) 15, and the media assignment-decision section 12, is a 
dialogue as a framework which performs such processing, for example 
The approach using the state-transition model like the voice dialog 
system indicated to JP.5-21 661 8,A, Sacerdoti, E.D. work "AStructure 
for Plans and Behavior", American It is realizable by using a hierarchy 
plan composite system which will exist in Elservier ** and 1977. 
[0052] The dialogue Management Department 13 has the module called 
a dialogue planner, as the sign 131 was attached and shown, although it 
does not illustrate this dialogue planner 131 with two storage tables, the 
dialogue regulation storage dialogue primitive storage table 1 32 and 1 33, 
it has three storage components called the plan storage section, an 
initial target record slot, and a current target slot, and this builds and 
performs a dialogue plan in it. 

[0053] The above is the configuration and its function of this equipment. 
The flow chart of actuation of the dialogue Management Department 1 3 
which is the nucleus module of this equipment is shown in drawing 19 . 
Based on this drawing, the usage of the principle of operation of this 
equipment and data is described. 

[0054] First, if this equipment is started, the dialogue planner 131 of the 
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dialogue Management Department 13 will develop all dialogue plans in 
order in the plant storage section which is not illustrated in the dialogue 
Management Department 13 based on ejection and its target from the 
initial target record slot to which the initial target for plan generation is 
set beforehand. 

[0055] One item of the plan storage section consists of five slots, "Node 
ID", a "target", a "condition", a "subnode ID list", and a "result", and 
calls this a plan node. If one certain target is set up, one plan node will 
be generated, the set-up target will be substituted for a target slot, and 
a unique notation will be assigned to a node ID slot at each plan node. 
[0056] One of five of "a success" unsettledness and "during 
processing", "failure", and "retry" is substituted for a condition slot. ID 
of the plan node corresponding to the sub target developed from the 
content of the target slot is contained in the form of a list by the 
subnode ID list slot. 

[0057] The activation result of the target of a plan node for the slot to 
belong is contained by the result slot. Henceforth, for example, the thing 
of a "plan [ under which the value of a node ID slot is pOOO ] node" is 
only described to be "a plan pOOO" or the "subplan pOOO." 
[0058] In order to develop a certain dialogue target at the sub target, 
the dialogue Management Department 13 has the dialogue regulation 
storage table 1 32. Two or more items showing a dialogue regulation as 
shown in the dialogue regulation storage table 1 32 at drawing 2 are 
stored, and the each is called a dialogue regulation node. One dialogue 
regulation node consists of four slots, "Rule ID", a "target", a "usage 
condition", and a "sub target list." As for "Rule ID", a unique notation is 
assigned to each set talk regulation like the node ID of a plan node. 
[0059] The target which the regulation attains in each set talk regulation 
node is dedicated to a target slot, the sequence of the sub target 
(generally plurality) for a target to be developed and generated is 
dedicated to a sub target list slot, and the conditions for applying a 
regulation are dedicated to a usage condition slot, respectively. 
[0060] The dialogue planner 131 searches for the dialogue regulation 
stored in the dialogue regulation storage table 132 sequentially from a 
top, and applies the first dialogue regulation which suits conditions. 
Generation and activation of a dialogue plan are realized by repeating 
recursively actuation of searching for and applying the regulation with 
which are satisfied of the given target out of a dialogue regulation, and 
developing and performing it at a sub target. 

[0061] And when an initial target is satisfied eventually, the whole 
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dialogue is successful and this equipment stops. Henceforth, for 
example, the value of a rule ID slot only describes the thing of "the 
dialogue regulation node which is rOO"" to be the "dialogue regulation 
rOO" or "a rule rOO." 

[0062] The dialogue planner 131 dedicates and manages the target of 
the dialogue in a certain event into a current target slot. Before 
searching for the dialogue regulation which develops the target of the 
present target slot, the dialogue planner 131 looks for the dialogue 
primitive storage table 133 which contained the dialogue primitive which 
is the smallest unit of the dialogue in which immediate execution is 
possible as shown in dra win g 3 . 

[0063] A dialogue primitive node, a call, and this consist one item of the 
dialogue primitive storage table 133 of three slots, "FAKUTO ID", a 
"target", and a "specification." When the content of the current target 
slot can simplify to the value of the target slot of a certain dialogue 
primitive node in the dialogue primitive storage table 133, dialogue 
PURANAN 131 performs the content which carries out the average of 
the target to a dialogue primitive, and is described by the corresponding 
specification slot of a dialogue primitive node. 
[0064] The primitive of the dialogue of the inter module which the 
dialogue Management Department 13 manages and performs is either of 
the input/output request to the inquiry, the reply and the setting-out 
command to I/O media components and input/output command of 
suitable media assignment to the media assignment-decision section 12, 
and the problem-solving means 1 5. 

[0065] The thing of the dialogue primitive node [ henceforth, ] of a 
"FAKUTO ID slot whose value is "f00"" is only described to be the 
"dialogue primitive fOO" or "FAKUTO fOO." 

[0066] When there are both a dialogue primitive which can be simplified 
at the target in the event of being at the time of expansion and 
activation of a dialogue plant, and a dialogue regulation, the target goes 
wrong. When a certain target goes wrong, the dialogue planner 131 is the 
approach same with backtracking a calculating machine in the activation 
process of the "PROLOG language" which is one of the computer 
program language, and performs back track of the expansion process of 
a dialogue plant, and retry of a dialogue regulation. 
[0067] Namely, a certain dialogue target Pi Activation is faced and it is 
the dialogue regulation Ri. When it applied and fails, it is dialogue planner 
131 Pi. It searches for other applicable dialogue regulations or a dialogue 
primitive, if there is an applicable thing, the dialogue regulation or a 
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dialogue primitive will be applied, and it is Pi. The target to be satisfied is 
retried. 

[0068] Pi the case where applicable all dialogue regulations and dialogue 
primitives go wrong — the dialogue planner 131 — dialogue target Pi 
Dialogue target Pc which was successful immediately before up to — 
retracing one's steps — the same — the dialogue target Pc It retries. 
As long as failure of target execution continues, when back track of a 
target is generated continuously, their steps are temporarily retraced to 
an initial target and the initial target itself goes wrong, it means that the 
whole dialogue had gone wrong and this equipment stops. 
[0069] The media assignment-decision section 12 is called from the 
dialogue planner 131 in the form of activation of the dialogue primitive 
"MA inquiry" which is an inquiry of multimedia allocation (MA). The 
media assignment-decision section 12 opts for suitable media allocation 
using the media [ degree ] assignment preference table 122 as indicated 
to be the condition of the media allocation in the event of the inquiry 
contained by the present media allocation storage slot occurring to 
draw ing 6 at the event. 

[0070] one item of the media [ degree ] assignment preference table 1 22 
— a media [ degree ] assignment preference node and a call — each — 
a media [ degree ] assignment preference node has four slots, "Node 
ID", the "present media allocation", "conditions", and "media [ degree ] 
allocation." 

[0071] The media allocation generation section 121 searches for the 
media [ degree ] assignment preference node which the conditions at 
the time of the content and the media assignment-decision section 12 
of the present media allocation storage slot being called suit out of the 
media [ degree ] assignment preference table 122, and generates the 
candidate of media [ degree ] allocation. 

[0072] Next, the media allocation generation section 121 searches for 
the mode-media allocation conversion table 1 24 as shown in drawing 7 
from the content of the media [ degree ] allocation slot of the 
discovered media [ degree ] assignment preference node, and verifies 
the constraint at the time of applying a certain media allocation. The 
node corresponding to mode-MEDEAA allocation, and a call and each 
node corresponding to mode-media allocation consist one item of the 
mode-media allocation conversion table 1 24 of four slots, "ID", a media 
allocation name", "constraint", and an "I/O mode group." 
[0073] In case the media allocation generation section 121 performs 
constraint verification of media allocation, it searches for the node 
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corresponding to the mode-media allocation which agrees with the 
media allocation name for which the slot of a media allocation name 
searches, and it verifies [ that it is / constraint / at the time of 
performing actually media allocation described by the constraint slot of 
the node corresponding to mode-media allocation / condition / in an 
ejection and inquiry event / satisfied in the constraint, and ]. 
[0074] If this constraint is satisfied, that media allocation candidate will 
be eventually chosen as media [ degree ] allocation, and the dialogue 
planner 131 will reply to him. In case the decision of media allocation is 
requested from the media allocation generation section 121 and it looks 
for a media [ degree ] assignment preference table here, when the value 
of the present media allocation slot of a certain media allocation node or 
a condition slot is the value of the slot is not used for verification. 
[0075] For example, the media [ degree ] allocation node whose content 
of the present media allocation slot is "-" can be adapted, if the content 
of the condition slot and the content of the constraint slot of a mode- 
media conversion table conform whatever the value of the present 
media allocation storage slot may be. Moreover, into the present media 
allocation slot and media [ degree ] allocation slot of a media [ degree ] 
allocation node, it can describe also in the form of the group of input 
media and output media in addition to the content of media allocation. 
[0076] Moreover, it is possible to use a variable for these input media or 
output media. For example, although the present media allocation slot 
value is "(spoken language, XXX)" and the value of a media [ degree ] 
allocation slot has become "(pen character string, XXX)" at G^awingS in 
the media [ degree ] assignment preference node whose value of a node 
ID slot is "m06" A media [ degree ] assignment preference node is 
applicable if it is a "spoken language", the input media, i.e., the input 
media by which current setting out is carried out, of the media allocation 
which is the value of the present media allocation storage slot, and other 
constraint [ whatever the output media ] is satisfied. 
[0077] At this time, the value by the side of the output media of the 
present media allocation storage slot is temporarily assigned to the 
value by the side of the output media of the present media allocation 
slot, i.e., variable", "XXX", and that substitution result is reflected in the 
media [ degree ] assignment preference node to which that present 
media allocation slot belongs. 

[0078] And retrieval and verification of constraint of a media [ degree ] 
allocation slot are performed by matching with the value of the I/O 
media slot of a mode-media allocation conversion table. For example, 
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when the value of the present media allocation slot is "human", call 
conditions presuppose that the media allocation generation section 121 
was called by "input failure." 

[0079] In order that the media allocation generation section 121 may opt 
for media [ degree ] allocation, it looks for the media [ degree ] 
assignment preference table 122 first, but among the media [ degree ] 
assignment preference nodes in aLrawJng 6 , since, as for the media 
[ degree ] assignment preference node of "m01 " to "m05", conditions 
do not suit, the value of a node ID slot is not chosen. 
[0080] Next, although the value of a node ID slot tends to verify whether 
the media [ degree ] assignment preference node of "m06" can be 
adapted, since the value of the present media allocation slot of this 
media [ degree ] assignment preference node is described by in the form 
of the group of input media and output media, the media allocation 
generation section 121 reconfigurates the value of the present media 
allocation storage slot in the group of input media and output media first. 

[0081] therefore, the media allocation generation section 121 searches 
for drawing 7 , and the value of the media allocation slot of the node 
corresponding to mode [ in which the value of a node ID slot is 
"mm01 " ]-media allocation from the same "human" thing as the value 
of the present media allocation storage slot the media allocation from 
the value of the I/O mode slot of this node corresponding to mode- 
media allocation — "human" input media — a "spoken language" mode 
simple substance — it is — output media — "a spoken language" — it 
is the combination in the mode and "face image" mode — " (a spoken 
language — ) It acquires that it is the I/O media group spoken language + 
face image." 

[0082] The media allocation generation section 121 is the value of the 
present media allocation slot of the media [ degree ] assignment 
preference node of the affair in which the value of Node ID is m06 using 
this information, "(spoken language, XXX)" and simplification are 
attained and a "spoken language + face image" is temporarily 
substituted for Variable XXX. 

[0083] This substitution is reflected in the media [ degree ] allocation 
slot of the same media [ degree ] assignment preference node, and the 
value of this slot becomes "(pen character string, spoken language + 
face image)." Next, the media allocation generation section 121 verifies 
the value "call condition = input failure" of the condition slot of the 
same media [ degree ] assignment preference node, and is successful in 
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this case. Furthermore, the media allocation generation section 121 
searches for the mode-media allocation conversion table like drawing 7 , 
and verifies constraint of the node corresponding to mode [ in which the 
value of an I/O mode slot is "(pen character string, spoken language + 
face image)" ]-media allocation. 

[0084] When that constraint is satisfied temporarily, selection of media 
[ degree ] allocation is successful, and the media allocation called "the 
pen character string input and human response" which is the value of 
the media allocation name slot of this node corresponding to mode- 
media allocation is substituted for the present media allocation storage 
slot, and it is returned to the dialogue Management Department 13 as a 
media allocation result (media allocation result). Termination of 
verification of the media [ degree ] assignment preference node to which 
the variable belongs cancels temporary substitution to Variable XXX. 
The thing of the media [ degree ] assignment preference node 
[ henceforth, ] of a "node ID slot whose value is "rnOO"" is only 
described to be "the media [ degree ] assignment preference node 
mOO." The thing of the node [ moreover, ] corresponding to mode [ in 
which a value is "mmOO" ]-media allocation of "ID slot" is only 
described to be "the node mmOO corresponding to the Mohd Media 
allocation." 

[0085] In order to make easy to understand [example of concrete 
actuation (first example of operation)] this invention, the example at the 
time of applying this equipment to the application of the hotel advice in 
a tourist resort as an example is described below. It is equivalent to the 
problem-solving means 15 which appeared by explanation of the 
application of the hotel advice in a tourist resort here, i.e., former. 
[0086] The dialogue primitive shown in the dialogue regulation and 
d rawing 3 which are shown in the dialogue regulation storage table 1 32 
of the dialogue Management Department 1 3 at drawing 2 is memorized 
beforehand. Moreover, the initial target of a "interactive session" is 
given beforehand, and is memorized by the initial target storage slot. 
[0087] When starting this equipment first, the dialogue planner 131 calls 
the regulation "rOO" with which it is [ an initial target ] satisfied of 
ejection and its target from an initial target storage slot. And it is 
developed as the dialogue planner 131 shows a dialogue regulation "rOO" 
to drawLng.4 . 

[0088] The dialogue planner 131 calls the user appearance which is an 
early sub target, a dialogue regulation "r1 1 " is developed, and a dialogue 
regulation "r1 2" is developed further. In order that the dialogue planner 
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131 may not have the dialogue regulation which can be adapted in the 
dialogue regulation storage table 132 in activation of a sub target "MA 
inquiry (standby, MAD", the dialogue primitive storage table 133 shown 
in drawing 3 is searched. 

[0089] Consequently, the dialogue planner 131 discovers a dialogue 
primitive "fOl", performs "a media assignment-decision section call" in 
the item of a specification, calls it to the media assignment-decision 
section 12 at the event, and conditions are suited and it asks available 
media assignment. 

[0090] The media assignment-decision section 12 opts for the media 
allocation which should lengthen the media [ degree ] assignment 
preference table 1 22 as shown in drawin g 6 based on the media 
allocation at the event, and the situation of available media or others, 
and should be chosen as a degree. In this case, a table element "m01 " 
is applied and media allocation of "standby" is chosen. 
[0091] Next, the media assignment-decision section 12 checks the 
constraint at the time of applying the media allocation which lengthens 
the Mohd-media allocation conversion table of drawing 7 , and 
corresponds. In this case, if the input sensor device is available, the 
media assignment-decision section 12 will choose media allocation of 
"standby" eventually, and will answer the dialogue Management 
Department. 

[0092] The dialogue Management Department 13 notifies the media 
which should be used for the input media processing section 1 13 and 
the output media processing section 143, respectively based on the 
media allocation which the media assignment-decision section 12 has 
answered. Thereby, the input media processing section 113 and the 
output media processing section 143 lengthen the Mohd-media 
allocation conversion table according to the advice, and an input/ output 
channel is set up by changing the condition of media selecting-switch 
SW and media distribution switch 143a, respectively. 
[0093] In standby media allocation of this example, an aperture and an 
output channel output message waiting like drawing 20 only for a 
personal sensor to a screen as an input channel. By the above 
processing, a sub target "p1 1" tends to be successful and the dialogue 
planner 131 tends to put the following sub target "detection standby" 
into activation. This sub target is a dialogue primitive as it is memorized 
by the dialogue primitive storage table 1 33 of drawin g 3 , and it is 
performed immediately. 

[0094] At this time, the condition of the plan storage section in the 
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dialogue Management Department 13 has become like drawing 5 . A 
dialogue primitive "user appearance" continues a waiting state until the 
condition of personal sensor appearance media changes. 
[0095] If a user stands in front of equipment, an input sensor will react, 
a dialogue primitive "user appearance" is successful, a subplan "p01 is 
satisfied, and the following sub target "MAR selection (first stage)" is 
called. This sub target is further developed by "r12", and like the time of 
selection of standby media, in order to perform a sub target "MA 
inquiry", the dialogue planner 131 calls the media assignment-decision 
section 12, and asks the media allocation for dialogue initiation. 
[0096] The media assignment-decision section 12 chooses the media 
allocation with which are satisfied of conditions and constraint by 
lengthening the media [ degree ] assignment preference table 122 and 
the Mohd-media allocation conversion table 124 which self holds like the 
case of standby media allocation. 

[0097] Consequently, in the case of this example, media allocation of 
being "human" is chosen, and it is answered by the dialogue 
Management Department 13. The dialogue Management Department 13 
is directing the media allocation to which it was answered in the input 
media processing section 1 1 3 and the output media processing section 
143, and performs the dialogue based on media allocation "Hugh Mann." 
[0098] this media allocation — that it is "human" is the media allocation 
which can carry out a natural dialogue with voice to an input by using a 
face image and a spoken language for speech recognition and an output. 
Therefore, that a user should just perform a question, want and a reply, 
a response, etc. in his voice, an equipment side will display and combine 
a face image with a display, answerback, a reply, etc. to it will be 
performed using a spoken language, and its human being is intelligible 
and tends to treat them. 

[0099] Furthermore, if data are given to the media assignment-decision 
section 12 from the input media processing section 1 13 or the output 
media processing section 143, for example as voice input Mohd or voice 
output MORT is unusable since a background noise is loud when above, 
media allocation "Hugh Mann" will not be chosen, but another media 
allocation will be performed, and a dialogue will be continued. For 
example, as not influenced at all of a background noise, it is a condition 
of the character representation according [ an output ] to a screen 
according [ an input ] to a keyboard. 

[0100] It becomes possible in this way to provide a user with the 
interface which can choose suitable I/O media now dynamically and is 
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easier to use them by having considered as the configuration which adds 
a means to opt for the media allocation for the basic configuration of 
conventional multi-modal dialogue equipment with suitable this invention 
by this according to a situation according to the situation of a dialogue, 
and an I/O means available at the event. 

[0101] Although the above is that a multi-modal dialogue equipment side 
chooses the optimal various I/O media dynamically corresponding to the 
occasional situation and the user was provided with the interface which 
is easy to use, it can also carry out as the configuration which offers the 
interface with which it is carrying out selection assignment, and each 
user tends to use for arbitration the thing for which a user side wants to 
use it of the various I/O media. 

[0102] What is necessary is just to add a means to put up the media 
which a user can further input to the above-mentioned configuration in 
multi-modal dialogue equipment, in order to make at arbitration the thing 
for which a user side wants to use it of the various I/O media the 
configuration which carries out selection assignment. By considering as 
such a configuration, it can know easily with what kind of means a user 
can input, and it becomes possible to offer the interface which is easier 
to use. 

[0103] For example, in media allocation "Hugh Mann" of the above- 
mentioned example of operation, by displaying the icon of a microphone 
on a screen like drawing 16 , a user is made to know that an input with 
voice is possible, and the effectiveness of urging an input with voice can 
be expected. Then, in the process of being satisfied with a dialogue 
regulation "r03" of a sub target "an initiation session", session initiation 
demand of a user and recognition of a system are performed, and it 
becomes possible to operate "A" and "B" of a scenario as an example 
which are shown in drawing 8 in this way. 
[0104] Namely, scenario A 

Thereby, in an advice system, a sensor detects this, turns ON a display, 
displays the title screen of a system, and displays on a screen the 
prompt one traveler stands in front of an advice system and which 
stimulates an input with a sound effect. 
[0105] Scenario B 

A traveler says with voice, "Please look for a hotel", and a system 
answers to this "The system understood." 

[0106] If the initiation session of such a content is successful, the 
dialogue Management Department 13 will announce that the task was 
started by the application which realizes the advice system which is a 
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problem-solving means. In this case, the application concerned which is 
a problem-solving means requires "acquisition of the initial retrieval 
conditions for database retrieval, i.e., the "date of arrival", a start day", 
"manpower", and a "budget" of the dialogue Management Department 
1 3 according to that advice. 

[0107] Based on this demand, the dialogue Management Department 13 
demands the decision of the again suitable media allocation for the 
media assignment-decision section 1 2. By performing the **** dialogue 
shown in "C" of the scenario of dra win g 8 using voice, displaying the 
condition of the retrieval conditions under input based on the answer 
like a screen as set media allocation "a variable display voice 
input/output" and shown in d rawin g 17 The dialogue Management 
Department 13 satisfies the demand of the application which is a 
problem-solving means. 

[0108] (Second example of operation) Second example ************ of 
operation which changed scene expansion next. By the dialogue to the 
scenario "C" of drawin g 8 , the initial condition of database retrieval is 
acquired, and a problem-solving means (background application) 
performs retrieval processing, and requires the output of a retrieval 
result of the dialogue Management Department 1 3 after retrieval 
termination. 

[0109] The outline of a retrieval result, i.e., the number of cases of a 
retrieval result and database elements in every affair, at this time, the 
problem-solving means 15 side with an output request Namely, a 
"name", "reading of a name", an "address", "magnitude", "the class of 
chamber and accommodation charges of each part store", In addition, it 
is announced that the "description", a "appearance photograph", a 
"map", "the way and time of concentration" to there, a "nearby 
station", and "a neighboring scenic spot" are outputted to the output 
media processing section 143 in the output section 14 of multi-modal 
dialogue equipment 10 from the problem-solving means 15 side. 
[01 10] At this time, the plan storage section of the dialogue planner 131 
is in the condition of having succeeded to the sub target "p42 of 
dr awing 10 . Although the dialogue planner 131 tends to search a 
dialogue regulation and it is going to apply "r07" first, in order to satisfy 
the "candidate decision" which is the following sub target "p43", since a 
usage condition does not suit, it fails, and other dialogue regulations are 
searched. 

[0111] Next, the dialogue planner 131 chooses "r08" and "a selection 
demand" which is a sub target "p43l" is developed. This sub target is 
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similarly developed by the sub target system train [p431 1, p4312, p4313] 
under a dialogue regulation "r1 0." 

[01 12] Among these sub target system trains, a sub target "p431 1 " is 
developed further, the plan storage section will be in a condition like 
drawing 10 , and 12 will be further called by the media assignment 
decision according to a dialogue regulation "r1 2." Although the same 
processing as old description is performed in the media assignment- 
decision section 1 2, since there are few retrieval results as three affairs, 
in retrieval of the media [ degree ] assignment preference table 1 22, a 
media [ degree ] assignment preference node "m3" is suited and chosen 
as conditions. 

[0113] Consequently, "touch panel display selection and element name 
voice reading raising" are chosen as I/O media allocation for a user to 
choose a candidate from retrieval results, and the media allocation is 
performed by the dialogue Management Department 1 3. 
[01 14] Like drawing 18 , about the accommodations of three affairs on a 
screen by this, respectively "A name", While the output media 
processing section 143 chooses the touch panel carbon button with 
which an "address", a "photograph", "magnitude", and "a neighboring 
name" have been arranged from the data poured from the problem- 
solving means 1 5 and sets it Voice will be outputted for the message to 
which the name of each housing is read out with voice and the selection 
is urged like the scenario "D" of dra wing 8 from a loudspeaker. 
[01 15] In the case of a certain dialogue equipment, the output gestalt 
same irrespective of the output number of cases was taken from the 
former, for example, there were quite a few cases where it became the 
I/O method which is hard to use unintelligibly for a user of being as 
displaying the retrieval result which has only several affairs conversely 
by the tabular format **** [, and ]. [ reading out the retrieval result of 
dozens of affairs or hundreds of affairs with voice ] 
[01 1 6] For example, also when there were many retrieval results of a 
database, with conventional dialogue equipment, it had occurred that the 
name of all housings is read out like the dialogue of the scenario "D" of 
drawing 8 . 

[01 1 7] This dialogue equipment has introduced the device in which the 
gestalt of I/O is dynamically set up according to a situation from the 
former in addition to the dialogue control mechanism based on a certain 
hierarchy plan generation method or a state-transition graph method 
that such a problem should be coped with. Therefore, when there was 
much object number of cases shown to a user, as according to this 
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equipment the dialogue Management Department 13 asked suitable 
media allocation to the media assignment-decision section 1 2 at the 
event, and it set up media allocation dynamically corresponding to the 
content of a reply, and it said that retrieval result presentation of a 
tabular format was chosen, for example, according to the occasional 
situation, retrieval result presentation can be carried out with the 
optimal gestalt. 

[01 18] For example, it sets in the dialogue of the scenario "D" of 
aVa wing 8 . In order to make a user choose the thing of a wish from the 
retrieval result of the accommodations which the presentation object 
number of cases of a retrieval result is three affairs as mentioned above, 
and are these three affairs I/O media allocation in the form which the 
node "m04" of the media [ degree ] assignment preference table 122 
was chosen in the media allocation generation section 121, consequently 
carried out to one affair like drawing 18 , and cleft the area of a screen 
comparatively was made. 

[01 19] However, when the retrieval result number of cases is ten affairs 
temporarily also in the same scene, I/O media allocation which does not 
give a user stress in the form where only the retrieval result number of 
cases is tell with voice is make, without set to "m05" the media 
[ degree ] assignment preference node choose in the media allocation 
generation section 121, and the screen output of a tabular format like 
drawing 21 be assign by this, and read out an accommodations name 
with voice. 

[0120] In this way, according to this equipment, corresponding to the 
informational quality and the informational amount which require an 
output, it is easy to accept application (problem-solving means) in a 
user, and it becomes possible [ setting up the combination of an 
intelligible output means ]. 

[0121] Furthermore, in order to give a big burden like [ in the case of the 
scenario "G" of drawingJS ], for example in case a user chooses a result 
also with the output gestalt in a tabular format when there is very much 
number of cases of a database retrieval result, it is not appropriate to 
show a retrieval result according to the demand of application (problem- 
solving means) itself. 

[0122] In this case, it is as follows. Now, like the case of the scenario 
"D" of above-mentioned drawing_8 , retrieval processing by the 
problem-solving means is performed, and the plan storage section of the 
dialogue planner 131 presupposes that the dialogue is progressing to a 
sub target "p431 \ \ " like dr awing 10 . 
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[0123] Since there is no media [ degree ] assignment preference node 
1 22 to which there are many retrieval result elements as 534 affairs, and 
conditions suit the media [ degree ] assignment preference table of 
dra wing 6 in the case of this example, the sub target "p431 1 1 ", i.e., "MA 
inquiry", to call the media assignment-decision section 12 will go wrong. 
In response to failure of "p431 11", the dialogue planner 131 performs 
the same relapse processing as the PROLOG interpreter which is one of 
the well-known computer languages. 

[0124] That is, it has a relapse into "MA selection (alternative 
presentation)" of "p431 1 " which is the high order gall of "p431 1 1 " first, 
and retries whether there is any dialogue regulation or dialogue primitive 
applicable to "p4311." in this case, the dialogue regulation "r12" which 
went wrong — "a selection demand" of "p43l" which "p4311" goes 
wrong and is the high order gall of "p431 1 " since there is also no 
unexpectedly applicable dialogue regulation and dialogue primitive — 
their steps are still retraced. 

[0125] Furthermore, since this "p43l" goes wrong similarly, their steps 
are retraced to the "candidate decision" of "p43", it is similarly 
searched for a dialogue regulation or a dialogue primitive applicable for 
retry of this gall, the dialogue regulation "r09" of drawing 2 is applied, 
and a sub target sequence [p4301 , p4302, p4303] is developed like 

drawi.Qg_J.l • 

[01 26] On the occasion of activation of "retrieval condition narrowing 
down" of "p4301", further, the dialogue planner 131 applies a dialogue 
regulation "r14", and generates a sub target sequence [p43011, p43012, 
p43013]. 

[0127] Next, the inquiry and problem-solving means 15 side reports a 
non-inputted variable list [the planned sightseeing site, a chamber 
format, and an accommodations format] for the variable which the 
dialogue Management Department 13 has not used for housing retrieval 
yet to the problem-solving means 1 5 to the request in the activation 
process of the "sheep input-variable inquiry" of the sub target 
"p4301 1 " which is a dialogue primitive. 

[01 28] A variable "the planned sightseeing site" is taken out from from 
among non-input-variable lists by activation of "member (an input 
variable, [the planned sightseeing site, a chamber format, an 
accommodations format])" of the following sub target "p43012." 
[01 29] And in order to make a user input it, using the variable as 
additional conditions, a dialogue regulation "r17" is applied to "the 
additional condition input" of the following sub target "p43013 , and a 
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sub target sequence [p430131, p430132, p430133] is generated. 
[0130] In the activation process of these sub target sequences, a 
dialogue like the scenario "G" of drawing 8 is performed, and this 
equipment becomes possible [ recovering failure in media allocation and 
continuing a dialogue with a user ] in this way. 

[0131] Drawing 1j shows the condition of the plan storage section when 
the utterance S14 of the system of the scenario "G" of drawing 1 1 
d rawing 8 is emitted. 

[0132] (Third example of operation) The third example of operation 
which changed scene expansion next is described. Suppose that the 
user found a route and time of concentration like the scenario "E" of 
d raw ing 8 after having a dialog to the scenario "D" of drawin g 8 . That 
is, since it did not have a means to perform I/O media allocation 
dynamically in conventional multi-modal dialogue equipment when a user 
asks for the output in different media allocation from the media 
allocation from the media allocation which dialogue equipment 10 opted 
for and outputted, it was impossible to have met the demand of such a 
user primarily. However, such a situation can be coped with with the 
equipment of this invention. 

[0133] When such a situation is encountered, this equipment can 
separate control of a dialogue, and assignment of I/O media, and can 
make media assignment preference of the flow of a dialogue of the 
media assignment-decision section 12 independently. Therefore, 
management in such a situation is enabled. 

[0134] When there was a demand of a user like U6 in the scenario "E" 
of dra wing 8 , the condition of the plan storage section of the dialogue 
planner 131 was in the condition of waiting for a user's input in the 
condition under processing of the "own alternative" of a sub target 
"p4313", like drawing 1 2 . Since a user's input was not the selection 
courtesy of a candidate but a re-output request in accordance with 
media allocation modification and it here, a sub target "p4313" goes 
wrong. 

[0135] Relapse processing occurs like the case of the second example 
of operation with this failure, "a selection demand" of "p431" which is 
the parent target of "p4313" goes wrong, and the dialogue regulation 
"r15" of drawing 2 is retried. The dialogue planners 131 are the 
conditions of "a user demand (output [a route, time of concentration]) 
in the activation process of this dialogue regulation, and selection of the 
suitable media allocation for the media assignment-decision section 12 
is requested. By this request, the media assignment-decision section 12 
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chooses the media allocation "a name, a photograph, and map touch 
panel display selection and route voice reading raising." And this 
selected media allocation is passed to the dialogue planner 131. 
[0136] While the dialogue planner 131 changes the output media 
allocation at the time of displaying the alternative of accommodations 
according to this selection, displaying that "name", a "photograph", and 
a "map" on a display about each accommodations of three affairs and 
setting up a touch panel carbon button, utterance [******** like S1 1 in 
the scenario "E" of drawing 8 is by bus as about 1 hour and a half. By 
train, from the place which went about 30 minutes, it changes to a bus 
and the countrified hermitage takes about 30 minutes. The Okuyama ** 
is in the place along which he walked about 30 minutes, after riding 
about 30 minutes by bus from here. The content " is uttered, a user is 
provided with information, and selection of the candidate of 
accommodations is again required of a user. The condition of the plan 
storage section in this event is shown in drawing 13 . 
[0137] In this way, according to this equipment, using the I/O approach 
which a user demands, the continuation of a dialogue in the form where 
the user had the leadership according to a user's interruption is attained, 
and it becomes possible to offer the interface which is easy to use for 
each user. 

[0138] (Fourth example of operation) the example of management 
actuation in a case since the content of the voice input obtained from 
the user was not right, as the system failed in speech recognition 
processing — next, it states as fourth example of operation, scenario "H 
of drawing 8 — " — since it did not have a means to perform I/O media 
allocation dynamically in conventional multi-modal dialogue equipment, 
like when a system fails in speech recognition processing, since the user 
changed the name of a place between readings although the system 
asked for the voice input from a user, it was difficult to recover 
handicap NYUKESHON with a user. 

[0139] However, if it searches for other dialogue regulations of being 
able to be satisfied with this equipment of the sub target to which the 
dialogue planner 131 caught failure in a dialogue with a user with the 
failure at the time of dialogue regulation activation, and failed in it also in 
such a case and there is an applicable dialogue regulation in it, it will 
retry. 

[0140] In this example, the viewpoint as a human machine interface of 
voice input has succeeded in ************ continuing a dialogue with a 
user by choosing and retrying from things input media called handwriting 
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recognition as failure occurred in the input media which are inferior in 
the viewpoint of the soundness of input process, especially the input 
process of the name of a place which a reading mistake tends to 
generate. 

[0141] Such control is realized as follows. As already stated, the 
condition of the plan storage section in the event of the system 
utterance S14, i.e., the utterance "whether it is the schedule which 
travels what kind of place", shown in gardenia fruit NARIO G of drawing 
8 has become like drawing 1 1 . 

[0142] Here, suppose that a user is going to nominate "Ryoan-ji" and 
changed between "************" and reading in the input process in 
spoken language Mohd. Supposing this "************ does not serve 
as an object vocabulary of speech recognition, speech recognition 
processing will fail in it. 

[0143] For this reason, a sub target "p4301331" goes wrong and "input 
process" of that parent target "p430133" is retried under another 
applicable dialogue regulation "r19." This "r19" is the dialogue regulation 
of asking for reinput by the same media allocation, and under this 
dialogue regulation, "p430133l" is developed by the sub target 
sequence [p43013301, p43013302] as shown in drawing 14 , and the 
dialogue of S15 and U9 is made between equipment and a user in those 
activation processes. 

[0144] that is, equipment:"** — I need your help once again — " 
User: "it is ************" 
The becoming dialogue is made. 

[0145] However, speech recognition processing goes wrong for the same 
reason, and a sub target "p430133" goes wrong again. Next, by making 
an input mistake a key, the applied regulation "r20" is a dialogue 
regulation which makes media allocation redo and reinput, and 
"p430133" is retried by this dialogue regulation and it is developed by 
the sub target train [p43013311, p43013312, p43013313]. 
[0146] A sub target "MA selection (input failure (planned sightseeing 
site)) of p430131 1"" is further developed by the sub target sequence 
[p430133111, p430133112] under a dialogue regulation "r12." 
[0147] "MA inquiry" of a sub target "p4301331 1 1" is a dialogue 
primitive, and the call of the media assignment-decision section 12 is 
performed according to the content of the dialogue primitive storage 
table 133. 

[0148] As already stated, with reference to the media [ degree ] 
assignment preference table 122 showing the media assignment-decision 
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section 1 2 in dra wing 6 , a media [ degree ] assignment preference node 
"m6" is chosen in this case. This media [ degree ] assignment 
preference node is selection of media allocation of changing only the 
input mode of the present media allocation into pen character string 
input mode from spoken language Mohd, and by continuing the dialogue 
of a variable input according to this media allocation, it goes ahead with 
a dialogue after S16 of drawing 8 , and it becomes possible to return to 
a dialogue which makes the task of selection of accommodations 
continue further. 

[0149] Namely, equipment: "it has not recognized. Please write the 
identifier of a destination with a pen." 

Character recognition is carried out because a user writes it to a screen 
as "Ryoan-ji" with a pen, and equipment recognizes "Ryoan-ji" to be 

-jj-* ^L* ^L* sl^ 

Equipment: "it is Ryoan-ji (************). Since it searches, please 
wait for a while." 

The becoming dialogue is successful. 

[0150] According to this equipment, failure is recovered by having 
considered as the configuration controlled to change an I/O means in 
this way that it should change to another I/O means and should carry 
out again to failure in a dialogue with a user, and it becomes possible to 
continue a dialogue with a user. 

[0151] furthermore , in case media allocation will be perform in the 
media allocation generation section 121 if the input mode components 
which add the device which catch external noise to this equipment , and 
judge the felicity of a voice input/output by the input be realize so that 
it may be cope with , when an external noise become intense , for 
example , be choose be lose about the media allocation describe that 
voice input be possible or that a voice output be possible by the 
constraint section of the Mohd-media conversion table of drawing 7 . 
[01 52] Consequently, while the media allocation which performs voice 
input is chosen and the input from a user is received with voice, when 
the noise becomes intense suddenly, [ for example, ] Although a media 
assignment-decision section call occurs with speech recognition failure 
like the above-mentioned example of operation and selection of media 
[ degree ] allocation in the media assignment-decision section is newly 
carried out, being chosen of media allocation which uses voice for I/O in 
the selection is lost. 

[0153] Since it is used in this way by having added the control which 
changes an I/O method according to change of an external- 



file://C:¥Documents%20and%20Settings¥saeko¥My%20Documents¥JPOEn¥JP-A-... 2007/02/15 



WW/ w w 



environment-element according to an external-environment-element 
according to the equipment of such a configuration, choosing the optimal 
I/O method, it becomes possible to continue a dialogue with a user. 
[0154] In addition, in an above-mentioned example, it has three modules, 
the input device processing section 111, the input mode processing 
section 112, and the input media processing section 113, about the 
configuration of the input section 1 1 . Although it has described having 
three modules, the output media processing section 141, the output 
mode processing section 142, and the output device processing section 
143, also about the output section 14 as a premise It is not limited to 
this and the configuration of the radial transfer components which 
realize this equipment can perform various deformation. 
[0155] Moreover, although the above-mentioned example has described 
as another component with which the input section and the output 
section became independent, realizing as the I/O section which unified 
these is also possible, and such an implementation approach is also 
within the limits of the meaning of this invention. 

[0156] Moreover, although the example mentioned above has described 
the media assignment-decision section as the component part which 
became independent of the dialogue Management Department or a 
problem-solving means, and components which became independent to 
the input section, the output section, the dialogue Management 
Department, etc. similarly about the problem-solving means The 
configuration approach of components of realizing this equipment is not 
what is limited to this. For example, it is possible to realize this 
equipment, even if there is no media assignment-decision section as 
components which became independent by realizing the dialogue 
Management Department with a media assignment-decision function. 
Moreover, for example, a problem-solving means is possible also for the 
function of the dialogue Management Department, the function of the 
media assignment-decision section, and constituting so that it may have 
the function of the input section, and the function of the output section 
further, and such an implementation approach is also within the limits of 
the meaning of this invention. 

[0157] Thus, the equipment which realizes this invention has the various 
implementation approaches about the bill of materials, and it refuses 
that it is not what is limited to the configuration of the components 
explained here. Moreover, although the example mentioned above has 
described as a premise using the media [ degree ] assignment 
preference table 122 which determines the next media allocation as the 
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media allocation at a certain event based on the situation in the event 
about the control system of the media assignment-decision section 1 2 
The media assignment-decision method which realizes this equipment is 
not what is limited to this. For example, it also becomes possible to 
realize selection of finer media allocation by being able to realize also in 
the media assignment-decision method based on a state-transition 
model, and adding slots and information, such as whenever 
[ preference ], to the media [ degree ] assignment preference table 122. 
[01 58] Moreover, although one batch, i.e., current media allocation 
information, is clearly described by the example mentioned above as 
history information of the media allocation used in case it opts for media 
allocation in the media allocation generation section 121, it is possible by 
describing the condition of the past media allocation, for example to the 
condition part of the media [ degree ] assignment preference table 1 22 
to use the history information of media allocation to the decision of 
media allocation. 

[01 59] Moreover, in an above-mentioned example, although the dialogue 
management based on a hierarchy plan generation method is described 
as a premise, the dialogue management method which realizes this 
equipment is not limited to this, and can be realized also in the dialogue 
control system based on a state-transition model. 
[0160] Furthermore, although the user is making the user know by 
displaying the available input approach with other information on a 
display in an above-mentioned example, the presentation format of the 
input media which realize this equipment is not limited to this, and can 
be realized also as a thing to show from other output devices, such as 
displaying on another display for example. 

[0161] Moreover, although the user is making the user learn only the 
available input approach by displaying on a display in an above- 
mentioned example Even if this equipment performs the response with 
voice to a user hard of hearing possible [ that making a user know is not 
limited to the input approach, and it displays similarly about an output 
method ] Since a user can know it, a user becomes possible [ requiring 
modification of media allocation ]. 

[0162] Furthermore, for example, the specific user model storage 
section is prepared in the media assignment-decision section, and it 
becomes possible to set up the optimal media allocation corresponding 
to each user, without a user requiring each time, when it receives a user 
hard of hearing [ above-mentioned ] because a user records beforehand 
available media or the media which cannot be used there. 
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[0163] Moreover, for example, record the situation of user each about 
the above media allocations on an adjustable medium like an IC card, for 
example, this equipment is equipped with the device in which IC card 
read is possible. By a user making that IC card insert and read using this 
device at the time of utilization initiation, and making the specific user 
model storage section memorize that read information Also in the 
equipment for many and unspecified persons, the situation of user each 
can be set up beforehand, and it can be coped with now, consequently 
the equipment which becomes possible from the event of setting out of 
the media allocation according to the situation of user each being the 
beginning of utilization initiation comes to be obtained. In this case, 
without carrying out demand actuation of modification, since it carries 
out from the event of each user's optimal media allocation being the 
beginning of utilization initiation, since smooth utilization is attained, it is 
useful to the improvement in service. 

[0164] Moreover, in being SHISUTEKE which only a specific user uses, it 
is made to perform specific user model storage to nonvolatile memory, 
and a user can enable it to update, or it can take for repeating utilization 
by the thing of giving a learning function and making it make a study 
result reflect the content of storage etc., and the interface which the 
user tends to use can be provided now. 

[0165] The interface which the user tends to use gradually can be 
offered only by a user not updating a user model by himself, but 
repeating utilization also for ** by equipping the media assignment- 
decision section with functional learning functions, such as ID3 and 
neuro, furthermore, and carrying out renewal of automatic of the content 
of the specific user model storage section by study. 
[0166] Moreover, although the specific user considered as available 
media or the media which cannot be used about the content of storage 
of the specific user model storage section above The content of storage 
of the specific user model storage section is not what is limited to this. 
For example, an interface desirable for each user can be similarly 
offered now by memorizing in the specific user model storage section 
about the media which a specific user selects, and such an 
implementation approach is also within the limits of the meaning of this 
invention. 

[0167] It is also possible to adopt the approach of making a user know 
with determining the output media for making the approach which is still 
more nearly available learn itself in the media assignment-decision 
section, and outputting using the output media allocation based on the 
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decision, and such an implementation approach is also within the limits 
of the meaning of this invention. 

[0168] In addition, although the above-mentioned example showed the 
example treating failure in the speech recognition by a user's reading 
mistake for example, like [ when an environmental sound (noise) 
becomes a cause and voice input is checked ] Also in a situation which 
injury generates in a dialogue by change of the situation of the media to 
be used The media monitoring function which supervises the condition 
of an input and output media is included in the media change section. It 
is set up by the media change section at a certain event, and the media 
allocation which is going to carry out current utilization is used without a 
failure, and it becomes possible by supervising whether communication 
with a user and dialogue equipment was attained safely to perform 
processing for clear trouble. 

[0169] Namely, for example, it sets in the scene where it receives the 
voice input from a user, the scene where voice tends to perform the 
output to a user, etc. When inhibition of communication with the user 
and dialogue equipment by generating of the loud noise etc. arises by 
monitoring perimeter noise level etc. continuously, for example, the 
dialogue Management Department is notified of this. For example, for 
example In a certain failure generating, stopping the I/O with voice and 
making it make a screen display and keyboard entry perform etc. makes 
a media change appropriately, or or — countering — etc. — the 
dialogue equipment which it becomes possible to realize processing of 
starting the sub dialogue for clear trouble, consequently can cope with it 
also to an environmental change is realizable. Thus, in this invention, it is 
possible to perform deformation various in the range which does not 
deviate from the meaning. 
[0170] 

[Effect of the Invention] As explained above, according to this invention, 
it becomes possible by having chosen suitable I/O media dynamically 
according to the available I/O means at the situation of a dialogue, and 
its event to provide a user with the interface which is easy to use 
according to a situation. Moreover, it can know easily with what kind of 
means a user can input, and it becomes possible to offer the interface 
which is easier to use. 

[0171] moreover, the quality and the amount of the information which 
application demands in an output — corresponding — a user — 
accepting — or [ ****** ] — being easy — it becomes possible to set 
up the combination of an output means, it becomes possible to offer the 
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interface which is easy using for the user of each in using the I/O 
approach which a user demands, and it becomes that it is possible to 
offer the interface which is easy using for each user by using a desirable 
I/O means for a user. Moreover, it becomes possible to offer the 
interface which is easy to use for each user by avoiding the I/O means 
which is not desirable for a user. Furthermore, failure in a dialogue with a 
user is recovered by changing an I/O means, and practically great 
effectiveness, like it becomes possible to continue a dialogue with a user 
is done so. 



[Translation done.] 
* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

"I.This document has been translated by computer. So the translation 

may not reflect the original precisely. 

2.**** shows the word which can not be translated. 

3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 
[Brief Description of the Drawings] 

[DrawLng J] The block block diagram for being drawing for explaining this 
invention and explaining the mode of concrete operation of this 
invention. 

[Drawing 2] Drawing in which being drawing for explaining this invention 
and showing the example of the dialogue regulation for explaining the 
mode of concrete operation of this invention. 

[Drawing 3] Drawing in which being drawing for explaining this invention 
and showing the example of a dialogue primitive storage table for 
explaining the mode of concrete operation of this invention. 
[Drawing 4] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (at the time of 
starting) for explaining the mode of concrete operation of this invention. 
[Drawing 5] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (at the time of 
standby) for explaining the mode of concrete operation of this invention. 
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[Drawi n g 6] Drawing in which being drawing for explaining this invention 
and showing the example of the media [ degree ] assignment preference 
table for explaining the mode of concrete operation of this invention. 
[ Drawin g 7] Drawing in which being drawing for explaining this invention 
and showing the example of the Mohd-media allocation conversion table 
for explaining the mode of concrete operation of this invention. 
[Dr awing 8] It is drawing about the example of the example scenario of 
operation which is drawing for explaining this invention and is assumed 
for explaining the mode of concrete operation of this invention. 
[D rawi ng 9] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (at the time of an 
initial condition input request) for explaining the mode of concrete 
operation of this invention. 

[ Drawing 10] Drawing in which being drawing for explaining this invention 
and showing **** of the plan storage section (at the time of initial 
retrieval termination extract) for explaining the mode of concrete 
operation of this invention. 

[Drawing 1 1 ] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (a condition 
narrowing-down dialogue, extract) for explaining the mode of concrete 
operation of this invention. 

[Drawing 12] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (at the time of a 
user interrupt extract) for explaining the mode of concrete operation of 
this invention. 

[Drawin g 1 3] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (at the time of user 
interrupt management extract) for explaining the mode of concrete 
operation of this invention. 

[Drawi n g 14] Drawing in which being drawing for explaining this invention 
and showing the example of the plan storage section (dealing with 
speech recognition failure, extract) for explaining the mode of concrete 
operation of this invention. 

[Drawing 1 5] The detailed block diagram for being drawing for explaining 
this invention and explaining the mode of concrete operation of this 
invention. 

[Drawing 1 6] Drawing in which being drawing for explaining this invention 
and showing the example of a screen of the media allocation Hugh 
Mann" for explaining the mode of concrete operation of this invention. 
[Dra win g 1 7] Drawing in which being drawing for explaining this invention 
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and showing the example of a screen of the media allocation a variable 
display voice input/output" for explaining the mode of concrete 
operation of this invention. 

[Drawing 18] Drawing in which being drawing for explaining this invention 

and showing the example of a screen of the media allocation "touch 

panel display selection and element name voice reading raising" for 

explaining the mode of concrete operation of this invention. 

[Draw ing 19] Drawing in which being drawing for explaining this invention 

and showing the example of an algorithm of the dialogue management for 

explaining the mode of concrete operation of this invention. 

[Drawing 20] Drawing in which being drawing for explaining this invention 

and showing the example of a standby condition for explaining the mode 

of concrete operation of this invention. 

[Draw in g 21 ] Drawing in which being drawing for explaining this invention 
and showing the example of a screen of the media allocation "tabular 
format touch panel display selection / retrieval result number-of-cases 
voice output" for explaining the mode of concrete operation of this 
invention. 

[Description of Notations] 

10 — Multi-modal dialogue equipment 

11 — Input section 

12 — Media assignment-decision section 

13 — Dialogue Management Department 

14 — Output section 

111 — Input device processing section 

112 — Input mode processing section 

113 — Input media processing section 

121 — Media allocation generation section 

122 — Media [ degree ] assignment preference table 

123 — The present media allocation storage slot 

124 — Mode media allocation conversion table 

131 — Dialogue planner 

132 — Dialogue regulation table 

133 — Dialogue primitive storage table 

141 — Output media processing section 

142 — Output mode processing section 

143 — Output device processing section 
143a — Output media distribution switch 
143b — Output media processing section 
143c — The Media Mohd interface 
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SW — Media selecting switch 
IDVa-IDVn — Input device 
ODVa-ODVn — Output device 



[Translation done.] 
* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation 
may not reflect the original precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 
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[Drawing 16] 
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[Drawing 8] 
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[Drawing 13] 
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